This is the chess half of the harness I built for CMU’s 11-768. The harness itself, and the bugs that taught me the most, are in The Harness Is the Product.
The opponent is a deterministic engine that looks two plies ahead: its own move and White’s best reply. I played White against it twice.
Before: the model picks, a tool plays
The model sees the board as text, picks a move from the legal list, and calls play_move. That was its only tool. It made 36 calls for 33 moves. Three were rejected as illegal rook moves, even though the legal moves were listed in front of it.
The game turned in a few moves. On move 14 the model played Qa6, and Black took the queen with a rook. The two-ply scorer I used to rank moves put Qa6 31st of 35 legal moves, and it cost nine points of material in one turn. By the end White had only its king.
After: a deeper search behind the same game
Then I replaced the model with a program: an alpha-beta search that scores every legal move and plays the best one. Same bot, same server.
- Draw
- depth 2, after 17 moves
- Win in 53
- depth 3
- Mate in 17
- depth 4
Depth 2 sees as far as the bot does, and it drew. One extra ply was enough to win, and a second one made the win quick.
Material stayed within two points of even the whole game, so the win came from a mating attack. It ends Bb6+ Qxb6 Qxf8#: the two-ply bot took a bishop sacrifice, and White’s queen mated. Seeing that line takes four plies, which a two-ply search does not have.
I added a bonus to the evaluation for pushing the enemy king toward the edge and expected it to matter. It did not. Without it, depth 4 still mates in 17, and depth 3 wins in 26 moves instead of 53. The game is deterministic, and re-running depth 4 reproduced it move for move.
What this does and does not show
Every move in the winning game came from my search program. Giving an agent somewhere to compute changes what the system can do without changing the model at all. It does not show that the model can write or run that search itself. Each result is one game against one deterministic opponent.
The next experiment
The harness already has what the model would need to run the search itself: a run_python tool, a simulate_move tool that plays out a move without changing the real game, and skills that load instructions on demand.
Model with play_move only: lost in 33. Search program alone: mated in 17.
Model with a search skill, run_python and simulate_move, choosing when to search and how deep.
My guess is that the model wins far more often with those tools, but I have no data for that yet. I can name ways it fails just as easily: the model might write a wrong search or skip the skill (I have seen it skip a skill before), running code costs steps and tokens that a step limit or compaction could cut short, and a slow search could hit the sandbox timeout.
To test it I would run a batch of games with and without the skill, with the same model and settings, and count wins, moves to mate, tokens used, and how often the model reached for the skill. A loss would still tell me something: whether the bottleneck is the model, the skill text, or the tools.