yceffort
◆ SERIES 4 POSTS

Deciding Whether to Swap the Model Behind a Coding Agent

Examining how SWE-bench builds and grades its tasks, designing an experiment to compare models in Claude Code, and deciding whether to switch based on resolved rate and cost.

README

A record of finding out whether the model used in Claude Code can be replaced with an open-weight model. Part 1 analyzes how SWE-bench tasks are built and graded and sets out the principles for comparison. Part 2 builds 36 tasks and a grader from 20 internal PRs and sets up the Claude Code run environment. Part 3 decides whether to switch models based on resolved rate, cost, and patch defects, and part 4 looks back on the 1,164 runs and what remains to think about.

04 ITEMS

All posts

in order
01

How to Compare the Models Behind a Coding Agent with SWE-bench

#ai
How do you measure the model performance of a coding agent?
Updated
28 min read
02

Building 36 SWE-bench Tasks from 20 Internal PRs

#ai#testing
Turning merged PRs from an internal WebSocket package into coding-agent evaluation tasks, validating the grader, and isolating personal instructions from the Claude Code run environment
Updated
22 min read
03

What Happens When You Swap the Model in Claude Code for GLM 5.3

#ai
I solved 36 internal tasks three times each with Claude Sonnet 5.5 and GLM 5.3. Comparing resolved rate, limit stops, cost, and patch defects across 216 runs, looking at how the conditions differ from public benchmarks, and deciding whether to swap the model
Updated
32 min read
04

Wrapping Up the Coding Agent Model Swap Experiment

#ai
Looking back on 1,164 runs and the process of building an internal benchmark. Cost per task, the Claude Code run environment, what I had hoped for from open-weight models, and the conditions I would change next time
7 min read