All postsSHORT6 min read
AI Coding Agents: Where the Cost Went in Runta's Test
Runta's test put 64.5% of scored-run list-price cost into runs that didn't pass. What it tested, its limits, and how to test your own agent setup.
AI coding agents: in Runta's test, 64.5% of the list-price cost of scored runs went to runs that didn't pass. Changing the harness, the coding-agent tool around the model, also changed what a task cost. Think Codex CLI or Claude Code. The useful question is which pairing works on your team's actual tasks.
What Runta tested
Runta published FrontierHarness Eval v1.1 on November 17, 2026. Shilin Zhu wrote the post. Runta sells an execution layer for AI agents and partnered with inference provider Baseten for this benchmark. These are Runta's reported results. We haven't reproduced them.
Runta reports 22 configurations across 18 distinct harnesses, tested with 5 open models on 62 terminal tasks with 4 attempts each. Runta says that produced 27,280 agent runs and used 83B tokens. Runta calls it "the largest-scale harness leaderboard" and says the expansion was 76× the size of v1.0. Runta says task time limits ranged from 15 minutes to 8 hours.
Runta's configuration count includes different DeepSeek Harness tool profiles and different OpenCode versions. Its Figure 1 legend lists Hermes Agent, ZCode, Grok Build, Oh My Pi, KimiCode, Alma, Codex CLI, mini-SWE-agent, Maka, Claude Code, Pi, OpenCode v1, OpenCode v2, DSH Minimal, Cline CLI, MiniMax Code, GitHub Copilot CLI, DSH Creator, DSH Standard, DSH Code, Antigravity CLI and Exo. Only the configuration names come from that chart. The numbers here come from Runta's prose.
Runta says Baseten served all 5 tested models: GLM-5.3 (zai-org/GLM-5.3), GLM Flash (zai-org/GLM-5.3-Flash), DeepSeek (deepseek-ai/DeepSeek-V4.1-Flash-1017), Muse (meta-models/muse-spark-open) and Kimi-K3 (moonshotai/Kimi-K3). No Claude, GPT or Gemini model was tested. Claude Code, Codex CLI and GitHub Copilot CLI were programs in the test, running open models.
Runta says it used Harbor evaluation, each tool's default settings and maximum reasoning effort. Each trial got a fresh Runta runtime.
Where the cost went
In Runta's test, 64.5% of the list-price inference cost of scored runs went to runs that didn't pass. Runta splits that into 52.7% for failures and 11.8% for timeouts. That describes spending inside this benchmark. It doesn't estimate your bill or agent spending across the industry.
Runta also says log audits flagged 216 trials for reward hacking, including 163 online lookups. Runta assigned those trials a score of 0.
The tool changes the cost, and the cheapest changes with the model
With each model held fixed, Runta reports a highest-to-lowest tool cost ratio of 1.8 to 5.0 times per task. Runta's largest per-task gap was DeepSeek: Cline CLI cost 5.0 times as much as OpenCode v1 for 2.0 more pass-rate points. The cost-per-pass spread in Runta's test was 1.9 to 4.9 times. Keep those measures separate.
Runta found that a higher cost didn't buy a higher pass rate. In Runta's test, the rank correlation of cost per task and pass rate was −0.39 to +0.22, depending on the model. Runta says that with 4 of its 5 models, the most expensive tool per pass passed less often than the cheapest.
Runta's per-task ranking put Codex CLI 1st with GLM-5.3 and 16th with Kimi-K3. In Runta's test, the median tool moved 11 of 22 places by cost per task when the model changed.
The per-pass ranks tell a different part of Runta's result: Codex CLI was cheapest with GLM-5.3 at $0.803 per pass, but ranked 12th with Muse and 10th with Kimi-K3. Those aren't the per-task ranks above.
Across all models in Runta's test, OpenCode v2 had the lowest cost per task at $0.561. Runta puts OpenCode v2, Pi, Codex CLI and ZCode on its cost/pass-rate frontier, with ZCode reaching the highest aggregate pass rate at 61.9%. Runta says the other tools cost up to 1.6 times as much and passed 50.6% to 60.2% of tasks. Across all models, Runta reports Codex CLI at the lowest cost per pass, $0.937, and Antigravity CLI at the highest, $1.63. These are reported benchmark estimates at list prices, with uncertainty in the rankings.
Runta still found that model choice sets most of the cost: in its analysis, the model accounts for 92.8% of the variance in average cost per task, measured on a log scale across the 110 harness-and-model pairs. The agent program came next.
What drives cost: steps
Runta identifies steps as the main cost factor, ahead of prompt size, tool count and cache hit rate. Each step sends context again, according to Runta.
In Runta's test, mini-SWE-agent had the smallest first request at 882 tokens, yet took 38 steps per pass and ranked 17th of 22 by cost per pass. My reading: a small opening prompt alone doesn't tell you how expensive the full run will be.
Failed runs sounded confident
Among the 5 best-value tools selected in Runta's test, 68% to 82% of failed runs ended with a confident final answer. Runta reports 68% for Pi and 82% for Codex CLI. Runta describes this as about 3 in 4 failed trials and says few admitted the work was incomplete. That finding applies to the selected tools in this benchmark.
Runta suggests a check before the agent stops might save some of that cost, but explicitly says it didn't test the idea. My opinion: it's an idea to evaluate on private tasks. There's no measured saving to quote.
The catch
Runta ran this on its own product with open models from Baseten. Runta says another provider's prices and caching could change the cost ranks. Closed-model results could differ. Runta plans to add Claude Opus 5.5, GPT-6 Astra and Gemini 4 Carbon. Those are planned. None has been tested yet.
Runta says each pass rate has a 95% interval of about ±10 points. A gap of a few points is a tie. Only 1 of Runta's 5 best-value picks held in nearly every resample: Codex CLI with GLM-5.3 stayed first in 99.6% of Runta's draws.
Runta reports the other winner frequencies as Oh My Pi with Muse at 53%, ZCode with DeepSeek at 47%, Pi with GLM Flash at 42% and Alma with Kimi-K3 at 25%. Runta calls these the most likely choices among several close tools. I wouldn't treat this as a settled universal ranking.
The story also drew discussion on Hacker News. The linked item showed 375 points and 126 comments when fetched on November 18, 2026. That discussion doesn't independently validate Runta's results.
How to test your own setup
Runta's advice is to test candidate tools with your model on a private dataset of your own tasks, large enough to see past the noise. Reading its benchmark is free. Running your own evaluation costs whatever your model provider charges for those runs.
My suggested starting workflow:
- Pick 2-3 agent tools and keep the model the same.
- Use your own already-solved tickets. Write pass/fail checks before running the agents.
- Run the same tasks through each setup. Keep failed runs and timeouts in the report.
- Compare pass rate and cost per pass. Record steps so you can inspect why a setup costs more.
- Re-test when you switch models, and check provider pricing and caching assumptions.
That workflow is my practical starting point. Runta's advice is to use enough private tasks to see past the noise.
The builder angle: my opinion
Sell dev teams a reusable private agent test built from their own solved tickets, with pass/fail checks and a report comparing agent setups on their work. The deliverable is their test set and the evidence for a tool choice.
I'd show which setup passes more for less on those tasks, while keeping uncertainty visible. This offer follows Runta's testing advice. It's my opinion, with no income guaranteed.
Daily email
Get each breakdown in your inbox.
One email each evening, right after the Short goes live: the AI tool it covers, the full write-up with sources, and how to get paid for it. Free, and you can unsubscribe anytime.