Back to School

At the end of Part 1, Qwen3.5-9B was stuck.

I had given it every helping hand I could think of. It had to describe its walls before moving. The game told it which exits it had already used. In the end it wasn’t even allowed to choose a wall. It still only ever got one agent out of the maze, the one that started five moves from the exit.

The frustrating part was that it did everything it was told. Asked to note every junction it passed, it wrote 125 notes in a game. It just never read them back to decide where to go.

So I stopped telling it what to do, and started teaching it.

Prompting versus fine-tuning

Everything in Part 1 was prompting: writing better instructions and sending them with every request. The model reads them, and they shape what it does next. But nothing about the model changes. Next turn, it starts from scratch and reads the instructions again.

Fine-tuning changes the model itself. A language model is billions of numbers, its weights, and everything it does comes from them. Fine-tuning shows it thousands of examples (here’s the input, here’s the answer you should have given) and nudges those numbers each time, until producing the right answer becomes the model’s own habit rather than something it’s reminded of.

The difference is roughly the one between handing someone a set of directions and having them practise the route until they could walk it in their sleep.

You don’t have to retrain all of it. I used a technique called LoRA, which freezes the original model and trains a small set of extra weights that sit alongside it. Here that was about 30 million numbers, under 1% of the model. It’s a bit like adding a specialised attachment rather than rebuilding the whole machine, and it’s what makes training possible on a home graphics card.

The catch with fine-tuning is the examples. You need a lot of them, and they have to be right.

A teacher that never makes a mistake

For most fine-tuning, that’s the expensive part. People label data by hand, or use a bigger, more expensive model to generate it.

I didn’t need either. A maze has a well-known, boring, reliable way out: depth-first search. At every cell, take an exit you haven’t tried. If there isn’t one, go back the way you came. In a maze without loops, that always finds the exit eventually.

So my teacher was a few lines of code, not a model. I played it through games on mazes of four different sizes, and recorded every turn as a training example: this is exactly what the model would have seen, and this is what it should have said back. A game took a fifth of a second. The whole dataset took under a minute and cost nothing.

There was one strict rule. The teacher could only see what the student could see. The game engine knows the whole maze; the model only sees its own cell, its coordinates and its notebook. If the teacher used the map, the model would be learning to copy decisions it had no way to make.

That rule created the interesting part of the problem. Most of depth-first search needs no memory: take an untried exit, back out of a dead end, keep going along a corridor. But when you come back to a junction you’ve fully explored, you have to know which way you originally came in. Nothing on screen tells you that. You have to have written it down.

So the teacher keeps one line per junction in its notebook, written the first time it arrives:

junction [3,-2] back W

This is the exact skill Qwen didn’t have in Part 1. It wrote plenty of notes; it never navigated by them.

First attempt: perfect moves, no memory

I trained Qwen3.5-4B on my RTX 3070 (8 GB, with Windows already using 1.3 GB of it), converted it for Ollama, and put it back in Part 1’s maze.

For the first 80 turns, it looked like a different model:

It matched the teacher’s move 124 times out of 125. And for the first time in the whole project, a local model got a second agent out.

Then I gave it 300 turns. The team covered 99% of the maze, and still only one agent escaped.

So I checked its notebooks against what the teacher would have written. Thirty junction notes were due. It had written none. It had learned every rule for moving, and nothing about remembering. Without its junction notes, every fully explored junction was a coin toss, and in a long game that’s most of the search.

The word that wouldn’t learn

My first suspicion was the plumbing. Training happens in one tool, and the model is converted, compressed and served through others. Any of those steps could change its behaviour. So I worked back through them: different sampling settings, no output constraints, the uncompressed model, and finally the training checkpoint itself, untouched.

Every one of them told the same story. The model had never learned it.

So I measured how surprised the model was by each word of the teacher’s answer on turns where a note was due. Almost every word scored close to zero: it knew exactly what was coming. Except one.

"... untried S,W; new junction, noting back E; go S"
                  ^^^

The word new. Its surprise score was around 3.7, roughly twenty times higher than the next hardest word.

That one word is the whole decision. Writing “new” means I’ve noticed this is a junction I haven’t recorded. Once the model writes it, the rest of the note follows easily. If it doesn’t, the note never happens. Every other word had been learned; the entire failure came down to a single token.

The reason it was hard: to write it, the model had to combine two facts it never wrote down anywhere. This cell has three or more exits. And I’ve only just arrived here for the first time. It was expected to work that out silently, and it had only seen about 60 examples of doing so.

There was a second lesson hiding here. During training, the loss (the number that tells you how wrong the model still is) had read about 0.00003 for most of the run. That says “nothing left to learn”. It was wrong by about a factor of a hundred, and the behaviour that mattered most was the one it hid.

Second attempt: show your working

Every reply already starts with a short check line, a scratchpad the model writes before it decides. In the first attempt, the note decision was buried in it as one leap. So I changed what the teacher writes there, to spell out each step:

before:  N brick, E open, S open, W open; arrived from E; used E; untried S,W;
         new junction, noting back E; go S

after:   N brick, E open, S open, W open; 3 open: junction; arrived from E; used E;
         untried S,W; notebook: no junction [-1,0], noting back E; go S

Now nothing has to be inferred silently. Count the exits: three, so it’s a junction. Look in the notebook: no line for this cell, so write one. And when there is a line, the check quotes it back, so the model practises reading its notes on every revisit, not just on the rare turns it has to backtrack.

It’s the same trick that helps people with maths: show your working. And I gave it ten times as many examples of writing a note as the first attempt had seen.

Results: seven of eight unseen mazes

Back in Part 1’s maze, 300 turns: all three agents escaped, on turns 5, 212 and 250. Those are exactly the same turns as the teacher’s own game. It had written 25 junction notes out of 25.

That’s the first time in this whole project that a team running on a local model has finished a maze.

It isn’t fast. Depth-first search is thorough, not clever, and one agent walked 250 moves to an exit 28 away. Claude Sonnet’s team cleared the same maze by turn 110. But Sonnet’s game cost $3.28 in API calls. This one ran on my own graphics card, and cost nothing.

The real test was mazes it had never seen. I ran eight overnight, from 8×8 to 15×15:

It hadn’t memorised mazes. It had learned the method, and in half of them it played the teacher’s game move for move.

The one that got away

The eighth maze is the most useful result in the post.

One agent arrived at a junction that also had a landmark on it, and the notebook had both on one line:

junction [7,7] back S; landmark helmet at [7,7]

The model didn’t recognise its own junction entry hidden behind the landmark, and wrote a garbled second note. A small mistake. But it led the agent, much later, into a situation the teacher never gets into: back at a fully explored junction, from the wrong side. There was no example of that anywhere in the training data.

With nothing to imitate, the model fell back on its most familiar pattern. It explored the same branch five times until the turns ran out. The teacher finished that maze in 123 turns.

That’s the known weakness of learning by imitation. A teacher that never makes mistakes never shows you how to recover from one.

What I learned

Teaching works where telling didn’t. Part 1 showed a small model will write a strategy down, quote it back, and not follow it. Fine-tuning a model less than half the size got it to play the strategy exactly.

A perfect teacher can be cheap. No hand-labelling and no big model: a few lines of algorithm generated every example in under a minute. If you can write down what good behaviour looks like, you can make your own training data.

One rare decision can sink everything. The first model was 99% right on moves and 0% right on memory, and no overall score showed it. Neither did the training loss. Only checking each kind of decision separately did.

Make the hidden step visible. The fix wasn’t more data of the same kind. It was writing the reasoning out, so the model learned a series of easy steps instead of one invisible leap.

Imitation doesn’t teach recovery. The model is only as good as the situations it has seen, and a teacher that never slips never visits the interesting ones.

And it all happened on a gaming PC: about five hours of training (more than half of it spent waiting while Windows quietly spilled graphics memory into system RAM), and not a single API call.

What it didn’t learn

The model learned exactly what the teacher did, and nothing else. The teacher searches alone: it never radios, never describes the exit from a landmark, never goes back for a teammate. So neither does the model. All three agents escaped, but as three people searching the same maze, not as a team.

The teamwork from Part 1 isn’t in this model. The obvious next step is a teacher that cooperates, plus recovery data so one slip doesn’t end the game.

The QLoRA training setup

For anyone who wants to try this, the short version:

• Teacher: depth-first search, written to see only the model’s own observation and notebook.
• Data: every turn of the teacher’s games, recorded with exactly the prompt the live game sends, byte for byte. A test plays a full game through the real game client to prove it.
• Training: Unsloth with QLoRA on Qwen3.5-4B, about 30 million trainable weights, on an 8 GB card.
• Serving: merged, converted to GGUF with llama.cpp, quantised to Q4_K_M (2.8 GB), and run in Ollama with the same settings as the original model.
• Gotchas: Ollama formats Qwen’s prompts with its own built-in renderer, not the usual chat template, so the training text has to be built to match it. On an 8 GB card under Windows, save checkpoints often and stop Ollama before training.

The dataset generator, training and export scripts, and the test that proves the training prompts match the live ones are all in the maze-agent repository.