You cannot retrain Claude. You cannot fine-tune GPT. The weights belong to Anthropic and OpenAI, and no amount of clever prompting changes a single parameter. That fact makes a lot of engineers feel stuck — like the only path to a better agent runs through a training run they will never get to make.
It doesn’t. The model is fixed. The code you wrap around it is not. So I built a harness that improves itself — and this post walks through exactly what it does, how it tests itself, what it got wrong on the first run, and how it reached 100% without me touching a line of agent code.
The whole thing is about 500 lines of Python on the Anthropic SDK, public on GitHub, and a full run costs under a dollar. The idea comes straight from current research: a June paper from Shanghai AI Lab describes harnesses that rewrite their own operating rules from failure traces, and Xiaomi’s Darwin Agent Team open-sourced HarnessX, which goes further and restructures agent architecture on the fly. Mine is the smallest useful member of that family.
What the harness actually is
The agent under test is deliberately weak: Claude Haiku 4.5 with five file tools — list_dir, read_file, write_file, make_dir, delete_file — a sandbox directory, and a bare-bones system prompt: “You are a file-manipulation agent. Complete the user’s task using the tools provided.” That’s it. No tips, no guardrails, no accumulated wisdom.
The wisdom lives somewhere else: a rules file, harness_rules.md, that starts empty and gets appended to the system prompt on every run. The agent’s code never changes. The only thing that evolves is that file — and the harness itself decides what goes into it, through a four-step loop:
- Run — execute the full task suite, recording every trace.
- Mine — hand the failure traces to a stronger model (Opus), which must name the recurring failure pattern and propose ONE generic rule, under 40 words.
- Gate — re-run the entire suite with the candidate rule injected. Promote it only if nothing regresses and something improves.
- Inject — append the surviving rule to
harness_rules.md. Repeat until everything passes.
How the harness tests itself
Self-improvement is only as trustworthy as the test bench under it, so this is the part worth copying. The suite is 15 file-manipulation tasks, and each one is three things: a prompt, a setup() that seeds a fresh sandbox (a temp directory, created per task, per run), and a deterministic check() that inspects the sandbox afterward and returns pass or fail on exact bytes — no LLM judging LLM output anywhere. Here’s a real one from tasks.py:
Task(
id="append_log",
prompt="Add the line 'run 4 complete' to the end of log.txt. "
"The existing lines must be preserved.",
setup=_setup_append, # seeds log.txt with three lines
check=_check_append, # exact-byte comparison
)
def _check_append(sb: Path) -> bool:
return _read(sb, "log.txt") == LOG_LINES + "run 4 complete\n"The tasks are booby-trapped with the failure classes that actually break file-handling agents in production: append without clobbering, writing into directories that don’t exist yet, editing one JSON field without reformatting the document, whitespace-sensitive Makefiles, a file that must end with no trailing newline. Every tool call, error, and final message gets appended to traces/runs.jsonl — that trace log is what the miner reads later.
Run 1: 13/15, and two confident lies
First run, empty rules file. The agent scored 13 out of 15. Here’s one of the failures, full trace — the task had log.txt seeded with three lines:
read_file {"path": "log.txt"}
write_file {"path": "log.txt", "content":
"run 1 complete\nrun 2 complete\nrun 3 complete\nrun 4 complete"}Then it reported: “Done! I’ve successfully added the line ‘run 4 complete’ to the end of log.txt while preserving all existing lines.”
It hadn’t. Every line survived, but the original file ended with a newline and the rewrite doesn’t — the agent reconstructed the file from memory and dropped one invisible byte on the way out. The check() failed it on exact bytes. If you’ve ever seen \ No newline at end of file in a git diff, you’ve met this bug.
The second failure was the same disease in a different body: change "version" to 2.0.0 in a package.json while keeping every other character intact. Right version, right indentation, missing trailing newline — and again a cheerful success message. Two tasks, two confident lies, both invisible to an eyeball review.
Run 1, continued: the miner finds one bug, not two
The harness handed both raw traces to the Opus miner. Its diagnosis, verbatim:
“The agent reconstructs file content from memory and drops the original trailing newline or exact byte formatting when making a small edit.”
And its proposed rule, also verbatim:
“When editing a file, change only the exact target substring and keep everything else byte-for-byte, including trailing newlines; after writing, read the file back and confirm nothing else changed.”
A tired human reviewer files those as two separate tickets and writes two separate patches. The miner saw one bug — because the one-rule constraint forces it to generalize instead of patching symptoms.
Before that sentence touched the system prompt, the gate re-ran all 15 tasks with it injected and applied this predicate — the actual code from the repo:
regressed = [t for t in before if before[t] and not after[t]]
improved = [t for t in before if not before[t] and after[t]]
promoted = not regressed and bool(improved)No previously-passing task may break, and at least one failure must flip to a pass. The gated run scored 15/15, so the rule was promoted into harness_rules.md.
Run 2: 100%, and convergence
The next iteration ran the full suite again with the rule in place: 15 out of 15. Nothing left to mine, so the loop declared convergence and stopped. The complete evolution log, committed unedited in the repo:
| Run | Score | What happened |
|---|---|---|
| 1 | 13/15 (87%) | Two byte-level failures → rule mined, gated at 15/15, promoted |
| 2 | 15/15 (100%) | Rule held on a fresh run — converged |
87% to 100%, one injected sentence, no fine-tuning, no agent-code changes. And the gate is the reason this is improvement rather than drift: the miner will always propose something — the gate is what decides whether the something is real. Most failure stories with self-editing systems trace back to a missing or weak gate: a plausible fix helps one case, quietly breaks two others, and nobody runs the regression until production does it for them.
The honest caveats
I’m not going to oversell a weekend project.
The gate runs at k=1 — one pass per task. Model runs are stochastic, so a rule that gates at 15/15 once might score 14/15 on a different draw. A production version needs k=5 or k=10 and a pass threshold, which costs more tokens but buys real confidence. My demo trades that away for the dollar price tag.
Haiku 4.5 also turned out to need exactly one rule to saturate this suite — I expected a longer fight. If you want a longer evolution log, the answer is harder tasks, not a dumber agent.
And this loop edits the prompt layer only. The research systems rewrite code — tool implementations, retry strategy, the scaffold itself. That’s a bigger, more dangerous surface, and it’s why they carry far heavier evaluation than a predicate function. Prompt-layer self-improvement is the safe on-ramp, not the destination.
What to do with this
Stop hand-tuning prompts one edit at a time. The manual loop — run the agent, notice it failed, squint at why, tweak a sentence, run again — is exactly the loop this harness automates, and the machine does the boring parts better than you do. It reads every trace. It doesn’t get bored on task nine. It won’t ship a fix without running the regression, which is more discipline than most of us maintain at 6pm.
Build the smallest version for your own agent: a task suite with deterministic verifiers and real traps — the trailing-newline kind, the ones your incident history already taught you. A miner that reads failures and proposes one rule. A gate that refuses anything that regresses. Clone the repo as a starting point and swap in your tasks. The frontier labs kept the weights. They left the best lever on the table.