A harness is the code around a model. It sends a prompt, parses the tool calls that come back, runs them, and feeds the results in again. It also owns the tool definitions, the memory, the skills and the stop conditions. For this course I wrote one from scratch and pointed a 4-bit Qwen3.8-27B at it, served from my home GPU server. Nothing here needed a frontier model.
Three layers
- 1Build the prompt: task, history, and a short catalogue of skills
- 2Ask the model for its next action
- 3Run each tool call and link its result to the call that asked for it
- 4If the prompt has grown past the threshold, compact the old part
Skills use progressive disclosure. The prompt lists each skill in a line, and an invoke_skill tool loads the full instructions only when the model asks for them. Compaction is the memory layer: when the prompt crosses a threshold, the model writes a working memory of the older part. The task and the most recent step stay word for word.
A bug fix that ran out of steps
The first job was a bug in the chess server that only showed up at the end of a game. The agent read the server code and the tests, wrote a reproduction, fixed it, and ran the suite: 9 passed. It used all ten of its allowed calls, and the budget ran out one call before the final message.
The agent did the work and the run still failed, because I had set the step limit without checking how long the task was.
A Django issue, and where the calls went
The second job was django__django-15368 from SWE-bench: bulk_update() mishandled a plain F() expression and wrote the text of the expression into the SQL instead of the column. The fix is one line.
- Calls 0 to 2Orient: list the repo, read the function, check the imports.
- Calls 3 to 4Patch: a sed one-liner failed on quoting, then a Python string replace applied the fix.
- Calls 5 to 33Verify: work out how to run Django's own test settings, reproduce the bug, run the queries suite.
- Calls 34 to 37Submit: a prose reply, invoke_skill, write the patch, send the final message.
- 5
- calls to find and apply the fix
- 29
- calls to check it
- 4
- compactions along the way
Most of the agent’s effort went into working out how to check its own work. A harness can help there with readable tool output that does not flood the prompt, and with enough steps to finish.
What compaction did to the prompt
I set the compaction threshold at 6,000 tokens. The harness compacted four times, and the prompt never went past 5,890 tokens.
| Compaction before call | Harness estimate | Server count, call before to call after |
|---|---|---|
| 15 | 5,352 to 2,006 | 5,642 to 2,986 (47% less) |
| 22 | 5,549 to 2,386 | 5,812 to 3,372 (42% less) |
| 29 | 5,202 to 2,648 | 5,387 to 3,736 (31% less) |
| 36 | 5,164 to 1,952 | 5,890 to 2,933 (50% less) |
My harness estimates tokens roughly, so I trust the server’s column. It is less flattering: the estimates suggested cuts of 50 to 60%.
The first working memory had the objective, the file and line to change, the edit already made, and the approaches that had failed: the sed command that broke on quoting, and test scripts that failed on app-label setup. Both survived the compaction, so they were still in front of the model afterward.
Five bugs that looked like the model
Tool results were only added on the error path. After a successful command the model saw nothing, so it acted as if it were working blind.
Every tool call gets a result linked to its id, whether it succeeded or failed.
- The skill nobody called. A submit skill existed, but the prompt only said skills were available. The model finished without it, no patch file was written, and the run failed. One required sentence fixed it: if a skill applies to finishing, invoke it before the final message. Even so, at call 34 of the Django run the model tried to wrap up in prose. The harness replied that no tool had been called, and the next call was
invoke_skill. I cannot tell how much of that was the prompt rule and how much was the nudge. - Tool schemas are code. Mine had unmatched braces, a set literal where a dict belonged, a missing key, and descriptions that lost their spaces when strings were joined. A tool description is part of the prompt, so a broken one breaks the prompt and nothing crashes to tell you.
- Parallel calls on shared state. A model can send two
play_movecalls in one response. The harness has to allow one move per turn and refresh its view of the game after any code the model runs. - The step budget from the chess-server fix above.
What I would tell the next person
- The model only knows what the tool results tell it. Test that path like a public API.
- Tool descriptions, skill descriptions and system prompts are the controls you have. Review them like code.
- If you want a behaviour, require it in the prompt. A skill being available does not mean it gets used.
- Budget steps for checking the work, not only for doing it.
- Compact old work, keep the task word for word, and never split a step from its results.
- Save every trajectory. That is how you tell a model problem from a harness problem.
The same harness also played chess. What happened when I put a search behind it is in Same Bot, Two Games.