A record of finding out whether the model used in Claude Code can be replaced with an open-weight model. Part 1 analyzes how SWE-bench tasks are built and graded and sets out the principles for comparison. Part 2 builds 36 tasks and a grader from 20 internal PRs and sets up the Claude Code run environment. Part 3 decides whether to switch models based on resolved rate, cost, and patch defects, and part 4 looks back on the 1,164 runs and what remains to think about.
◆ SERIES 4 POSTS
Deciding Whether to Swap the Model Behind a Coding Agent
Examining how SWE-bench builds and grades its tasks, designing an experiment to compare models in Claude Code, and deciding whether to switch based on resolved rate and cost.
README
04 ITEMS
All posts
in order
01
How to Compare the Models Behind a Coding Agent with SWE-bench
How do you measure the model performance of a coding agent?
02
Building 36 SWE-bench Tasks from 20 Internal PRs
Turning merged PRs from an internal WebSocket package into coding-agent evaluation tasks, validating the grader, and isolating personal instructions from the Claude Code run environment
03
What Happens When You Swap the Model in Claude Code for GLM 5.3
I solved 36 internal tasks three times each with Claude Sonnet 5.5 and GLM 5.3. Comparing resolved rate, limit stops, cost, and patch defects across 216 runs, looking at how the conditions differ from public benchmarks, and deciding whether to swap the model