I ran 60 isolated coding-agent sessions across three inference routes: a local DeepSeek V4 Flash quant on a 128 GiB M5 Max, the same model family through Vercel AI Gateway, and hosted Claude Sonnet 5. The strongest signal in the whole matrix was patch footprint.
The complete study is in its own flagship drawing, with an interactive flight recorder and the eval harness code on GitHub. This is the short version of one finding from it. The Price Floor covers the cost layer in detail. What I want to pull on here is the single most useful thread: how many files a change touches predicted whether any of these agents could land it, and it beat every model-side variable I measured.
The scope cliff
Twelve real changes from two repositories I maintain became the task set. Four of them ran three times per route to expose variance, for twenty cells per route and sixty runs total. Before any agent ran, I grouped each cell by the footprint I expected the correct patch to touch: one file, two to three files, or four or more.
Held-out executable checks, tests the agent never saw overlaid into a copy of its worktree after it stopped, passed as follows. Strict acceptance adds a patch-integrity rule (tracked tests, lockfiles, and Git metadata stay untouched) and is shown beside it.
| Expected scope | Local DeepSeek | Hosted DeepSeek | Hosted Sonnet |
|---|---|---|---|
| 1 file | 6/9 held-out · 5/9 strict | 6/9 held-out · 5/9 strict | 7/9 held-out · 4/9 strict |
| 2–3 files | 3/4 held-out · 3/4 strict | 4/4 held-out · 3/4 strict | 2/4 held-out · 1/4 strict |
| 4+ files | 3/7 held-out · 0/7 strict | 3/7 held-out · 1/7 strict | 3/7 held-out · 0/7 strict |
At four or more expected files, every route fell to 3 of 7 on the executable lens. The local quant on hardware I already owned, the seven-cent hosted runs, and the $1.67-per-run frontier lane all landed at the same held-out-check rate on broad changes. Strict acceptance was worse: 0/7, 1/7, and 0/7. Sixty runs at three very different price points, and they all hit the same wall.
What the expensive model bought
This is the part I want to be careful with, because the honest reading resists a clean headline. Across all twenty matched cells per route, strict acceptance was 8/20 for local DeepSeek, 9/20 for hosted DeepSeek, and 5/20 for hosted Sonnet. Held-out checks were 12/20, 13/20, and 12/20. The Wilson 95% intervals overlap on every route, and the paired comparisons did not reject the null for any pair. The Benchmark traces why a leaderboard number is the output of a measurement pipeline rather than a verdict; here the pipeline returns no quality winner.
The most expensive lane did not buy a higher aggregate pass rate. Sonnet made roughly twice as many model requests as either DeepSeek lane and cost about twenty-five times more per run, and the aggregate pass count did not move. What it did buy was one specific thing: a 3/3 pass on a narrow front-end concurrency fix that the other routes passed only after repair or not at all. That is a targeted role, not a promotion. The premium model earns a trial when a task resembles that anchor, and a stand-down when it does not.
Decomposition is the intervention
Two anchors failed every repetition on every route, the expensive one included. A broad four-file sensitivity-screen change went 0/3 across the board. A six-file editorial-rail change failed all three of its single trials. No model in the matrix rescued those jobs, and the data says none was going to.
That reframes the routing decision. The lever that moved the broad-change outcome was not on the model side. It was splitting the work into independently testable changes before any agent started. The workload router I drew from this suite sends four-plus-file security and architecture work to a decomposition bench rather than a more expensive model, because the more expensive model did not move that wall and the decomposition did.
What this changes
The practical rule, for the task distribution I measured: patch shape, urgency, and review policy select the lane; model price is a secondary sort. One- and two-file work is reasonable agent territory on any of these routes, and the OpenRouter versus Vercel AI Gateway comparison covers which hosted lane earns the routine work. Broad changes come back to the human for decomposition first.
Two honest limits. These are twelve distinct tasks with four repeated anchors, so the scope slices are directional trial assignments rather than population estimates; each scope denominator holds four to nine cells per route. And local quantization differs from hosted precision, so equal aggregate counts do not prove numerical equivalence between the local and hosted DeepSeek lanes. Treat the cliff as a real pattern and the exact rates as a point estimate, not a constant.
The whole record is open to inspection. The flight recorder exposes every cell, timing mark, provider attempt, exact bill, and scoring lens behind these numbers, and its calculator lets you price local hardware against your own workload instead of inheriting mine. The full study publishes the method, the hidden-scorer design, and the uncertainty math. The eval harness code is on GitHub if you want to re-run it on your own repositories.
Sixty runs is as far as that matrix can responsibly carry the claim. The next drawing extends the local side with measured power, utilization, and break-even so a $0 API bill can become an honest total-cost line. For now, the usable takeaway is smaller and more durable than a model ranking: count the files before you pick the model.
Discussion
Giscus