Claude Code runs fastest but costs 2.7x more
Four agent harnesses ran the same model on the same 30 tasks, and the price of a finished task varied by 2.7x.
Composio held the model fixed and swapped only the agent harness, isolating what the software wrapper itself costs. (Image: The Decoder)
AI tooling company Composio put DeepSeek V4 Flash inside Claude Code, Codex, OpenCode, and Oh My Pi, then handed all four the same 30 agentic tasks driving Gmail, GitHub, Slack, and Notion. Pass rates landed in a tight band of 14 to 17 out of 30, so no wrapper made the model meaningfully more capable. Cost and latency were not tight at all: a successful task ran $0.073 under OpenCode and $0.195 under Claude Code, while Claude Code's 123-second median finished 2.2x quicker than Oh My Pi's 272 seconds. The Decoder published the breakdown on August 6, 2026, and the practical reading is that picking a harness is a budget and latency decision rather than a capability one.
At a glance
· One model, four wrappers: DeepSeek V4 Flash across 30 agentic tasks using Gmail, GitHub, Slack, and Notion.
· Tasks passed: Oh My Pi 17/30, Claude Code 16/30, Codex 16/30, OpenCode 14/30 — a three-task spread.
· Cost per successful task: OpenCode $0.073, Codex $0.081, Oh My Pi $0.103, Claude Code $0.195.
· Median time per task: Claude Code 123s, OpenCode 130s, Codex 245s, Oh My Pi 272s.
· Seven of the 30 tasks passed or failed purely on which harness ran them.
One model, four wrappers
Most published agent comparisons swap the model and hold everything else roughly constant. Composio did the opposite. DeepSeek V4 Flash stayed fixed across every run, so any difference in the results traces back to the software wrapped around it.
That wrapper is usually called a harness: the loop that sits between the model and the outside world. It writes the system prompt, decides how tools are described, chooses how much conversation history gets replayed on each turn, decides when to compact or summarize, and sets what happens after a tool call fails. None of that shows up on a model card, and all of it shows up on the invoice.
The 30 tasks were not coding katas. They ran against Gmail, GitHub, Slack, and Notion — the kind of multi-step work where an agent has to read something in one system, make a judgment, and write to another. That matters, because tool-heavy work is exactly where harness design has the most room to help or hurt.
A three-task spread in pass rates
Harness
Tasks passed
Median time
Cost per success
Claude Code
16/30
123s
$0.195
Codex
16/30
245s
$0.081
OpenCode
14/30
130s
$0.073
Oh My Pi
17/30
272s
$0.103
Oh My Pi finished the most tasks at 17 of 30. Claude Code and Codex tied at 16. OpenCode came last at 14. In percentage terms the entire field sat between 47% and 57%.
DeepSeek V4 Flash across four agent harnesses on 30 tasks: pass rates cluster together, cost and latency do not. (Image: Composio via The Decoder)
Nobody cleared 17 of 30, and that ceiling is the more interesting half of the result. On this task set the limit was the model's ability to plan and recover, and no amount of harness engineering moved it much. If you are unhappy with your agent's completion rate, switching frameworks is unlikely to be the fix.
The spread also deserves a caveat the headline numbers hide. Thirty tasks is a small sample, and a three-task gap is roughly ten percentage points on a base of thirty — comfortably inside the range that could shift on a rerun. Read the pass-rate ranking as "roughly equal," not as a leaderboard.
Where the 2.7x price gap comes from
The cost column is where the field genuinely separates. OpenCode delivered a successful task for $0.073; Claude Code needed $0.195 for the same outcome. Run a thousand successful tasks and that is $73 against $195.
Claude Code used the fewest tool calls and generated the fewest output tokens of any harness tested — and still produced the largest bill.
That combination inverts the usual intuition, which is why it is worth sitting with. Output tokens are the expensive ones per unit, so the harness that writes the least should be the cheapest. It was not.
Composio's public figures do not break out input versus output tokens, so what follows is a reading rather than a measured finding. The money that is not going into output has to be going into input: the system prompt, the tool schemas, and whatever slice of prior conversation the harness replays on every turn. A large fixed preamble billed once per turn compounds quickly, and it compounds no matter how terse the model's replies are. Fewer, denser turns are not automatically cheaper turns.
What cost per successful task hides
The metric Composio reported divides total spend by the tasks that actually finished correctly, which means failed attempts are amortized into the price of the survivors. That definition changes how the ranking should be read.
OpenCode is the clearest case. It passed the fewest tasks of the four, 14 of 30, and still came out cheapest per success at $0.073. For that to work, its cost per attempt has to be low enough that sixteen failures barely move the average.
Claude Code shows the mirror image. It passed 16 of 30, two more than OpenCode, and still landed at $0.195 — a better hit rate could not rescue the average. Both readings follow from the arithmetic of the metric rather than from any per-attempt figure Composio published.
The consequence for planning is useful. Moving to a cheaper harness can lower your cost per success even if its pass rate drops slightly, because the two numbers do not move together. Track them as a pair, and always divide by successes rather than by API calls — cost per call would have hidden the retries, the failed runs, and the context overhead all at once.
Seven tasks that flipped
Seven of the 30 tasks passed or failed based solely on which harness executed them. That is a little under a quarter of the set, and it is arguably the most actionable number in the study.
It does not mean one framework quietly wins those seven. If it did, the pass rates would have separated, and they did not — the wins largely cancel out across the field. What it means is that harness choice is not only a tax on speed and price. On roughly a quarter of the workload it decides the outcome, in both directions.
The operational consequence is straightforward. If you evaluate a model inside one harness and then ship it inside another, expect a meaningful share of your borderline tasks to behave differently in production than they did in the eval. Benchmark inside the harness you actually deploy.
Choosing a harness on price and latency
For interactive work with a person waiting on the result, the split is clean. Claude Code at 123 seconds and OpenCode at 130 seconds sit in one group; Codex at 245 seconds and Oh My Pi at 272 seconds are roughly double the wait. Two minutes versus four and a half is the difference between a tolerable pause and an abandoned session.
For batch work that runs unattended overnight, latency stops mattering and $0.073 becomes the number to optimize. At volume, OpenCode's price advantage over Claude Code is the single largest lever in the data, and it costs only two tasks out of thirty in completion.
Claude Code occupies the awkward middle of the table. It is the fastest option and it does the most with the fewest tool calls, but it charges 2.7x OpenCode's rate for the same pass rate Codex delivers at $0.081. If it is already in your stack, speed is a defensible reason to keep it; unit economics is not.
Reproducing this on your own workload takes four steps: hold the model fixed and swap only the harness, connect the tools you actually use rather than a public suite, record pass rate alongside median wall-clock time and total spend divided by successes, and count how many of your tasks flip outcome between harnesses. If that flip share lands anywhere near seven in thirty, the harness belongs in your correctness review, not just your billing review.
One limitation before anyone rewrites their infrastructure: this is one model, thirty tasks, and a single point in time. Agent frameworks ship prompt and context-management changes constantly, and each of those changes moves exactly the input-token overhead that appears to drive the cost gap. Numbers this specific have a short shelf life.
The metric to standardize on
The four harnesses did not separate on whether they could do the work; they separated on what the work cost and how long it took. That turns framework selection into a procurement question your own workload can answer in an afternoon. Measure cost per successful task rather than cost per call, and measure it again after the next harness update.