What the model was thinking

Second part in a series on running LLMs locally — building the things people actually build, instrumenting them, and seeing what the measurements say.

Previous: What the model actually saw — context length and silent truncation

Marcus is a backend developer on a four-person platform team. Last week he developed a tool that used a local LLM to analyse some incident reports. Encouraged by the results, Marcus looks to scale it up before more documents come in.

He’s read that there is a newer version of the Qwen model he was using, and this one has a higher maximum context length, so would be able to handle more documents.

He downloads the new version:

ollama pull qwen3.5:4b

And then updates his constant to use it:

const string Model = "qwen3.5:4b";

He kicks off the tool and goes to make a cup of coffee.

When he comes back he checks the results:

### Incident Summaries

| Identifier | Duration | Root Cause |
| :--- | :--- | :--- |
| **INC-0417** | 3 hours 12 minutes | A refactored dispatcher lost a `using` declaration for its database context, causing connection pool exhaustion that led to event delivery backlogs due to an unadjusted wait timeout. |
| **INC-0463** | 47 minutes | An infrastructure migration omitted the manifest for a certificate renewal cronjob, resulting in expired certificates and regional outages because alerts were configured on job failure rather than absence. |
| **INC-0498** | 6 hours 5 minutes | A schema migration acquired an ACCESS EXCLUSIVE lock on our largest table during business hours due to lack of row count visibility in reviews, causing prolonged write contention. |
| **INC-0521** | 22 minutes | A release altered a feature flag default value while removing deprecated logic simultaneously, resulting in zero-priced quotes when the rating engine fell through due to silent fallback behavior. |

### Common Themes

All four incidents stem from failures in configuration management and monitoring blind spots that allowed "silent" or masked errors to propagate:
*   **Configuration & State Management:** Critical state was lost or mismanaged (missing `using` disposal, missing cronjob manifest, incorrect flag defaults) without triggering alerts.
*   **Monitoring Blind Spots:** Alerts were configured on the wrong metrics (p50 vs saturation, job failure vs absence), failing to detect connection pool exhaustion, certificate expiry, price distribution shifts, or lock contention until damage was done.
*   **Deployment & Review Process Risks:** Migrations and releases occurred during peak hours without maintenance windows; code reviews lacked visibility into system state (table size, inventory diffs) leading to unanticipated resource saturation or logic failures.

----------------------------------------
  FinishReason         : Stop
  InputTokenCount      : 4721
  OutputTokenCount     : 6147
  TotalTokenCount      : 10868
  ElapsedMilliseconds  : 74806

All four incidents summarised - check

Finish Reason is Stop - check

And the list of root causes - it’s been more specific about the common cause - “alerts were configured on the wrong metrics (p50 vs saturation, job failure vs absence)” - check

Input Tokens look ok - no silent truncation and his guard didn’t fire - check

Elapsed Milliseconds - 1 minute and 14 seconds. It was quicker than that on the old model - about 11 seconds.

And output tokens - 6147 - last time it was only 536. Eleven times last time’s output count, and the answer on screen is the same length. So where were all those tokens and time spent?

Thinking mode

Last time he ran an ollama ps command to see what models were running. Checking the docs he sees there is an ollama show command that gives information about a model.

He tries that for both the old and new models:

PS C:\Users\spenc> ollama show qwen2.5:7b
  Model
    architecture        qwen2
    parameters          7.6B
    context length      32768
    embedding length    3584
    quantization        Q4_K_M

  Capabilities
    completion
    tools

PS C:\Users\spenc> ollama show qwen3.5:4b
  Model
    architecture        qwen35
    parameters          4.7B
    context length      262144
    embedding length    2560
    quantization        Q4_K_M
    requires            0.17.1

  Capabilities
    completion
    vision
    tools
    thinking

  Parameters
    top_k               20
    top_p               0.95
    presence_penalty    1.5
    temperature         1

The newer model has two new entries under Capabilities - vision and thinking.

He decides to research what these are.

vision allows the LLM to process image data - interesting, but that’s not relevant here.

But thinking - that sounds like a possible explanation - but what does that mean?

In thinking modes, the model generates reasoning tokens before it starts on its answer. This is the model talking to itself and reasoning on what it is going to answer with. Qwen3.5 models default to thinking mode being on. That explains why there are more output tokens than he can see, and why it took longer. The model was doing extra work.

This is unexpected - nothing at any point told him. Not the pull, not the tag, not his code.

Time to start measuring

Marcus needs to find out if this is a good thing or not.

First task - is he able to turn this mode off?

The answer is, yes he can. The CompleteChat() function allows an options object to be passed in, and it has a few interesting properties.

var options = new ChatCompletionOptions() 
{ 
    Seed = 42, 
    Temperature = 0F, 
    ReasoningEffortLevel = ChatReasoningEffortLevel.None 
};

var resp = client.CompleteChat(chatMessages, options);

Seed allows the caller to set the random seed the model uses throughout - this is very useful when it comes to measuring; it ensures the same random sequences are followed each time.

Temperature is a dial that introduces some variation into the tokens the model generates. At temperature 0 it will always choose the highest probability next token. As you turn that dial up it allows it to introduce variance, at the cost of coherence if you push it too far.

And ReasoningEffortLevel - this is actually an enumeration of None, Low, Medium or High - so this is something else Marcus could measure.

Note: as of writing this, some of these properties in the SDK are still in early-access. You will need a #pragma warning disable OPENAI001 line in the code to “opt-in” to use them and suppress the warning

Marcus updates his code and gives it another run. First off he tries Low reasoning; maybe at low it will run quicker and generate fewer reasoning tokens. He gives it a go:

----------------------------------------
  FinishReason         : Stop
  InputTokenCount      : 4721
  OutputTokenCount     : 6147
  TotalTokenCount      : 10868
  ElapsedMilliseconds  : 75407

Hmm ok, still 75 seconds and still 6147 output tokens. Let’s try it with None:

----------------------------------------
  FinishReason         : Stop
  InputTokenCount      : 4723
  OutputTokenCount     : 310
  TotalTokenCount      : 5033
  ElapsedMilliseconds  : 12009

That looks better - the output tokens are back to a similar level as Qwen2.5 and elapsed time is back down to 12 seconds. And as for the content:

Here is the summary of the four incidents:

*   **INC-0417**: A 3-hour backlog occurred because a persistence layer refactor lost `using` declarations for database contexts, exhausting the connection pool and causing deliveries to queue rather than fail due to an unadjusted wait timeout.
*   **INC-0463**: An outage affecting two regions lasted 47 minutes when a management cluster migration omitted the manifest for a certificate renewal job, leading to expired TLS certificates that were only detected after they had already caused failures.
*   **INC-0498**: A six-hour write lockout happened because a schema migration on our largest table was coupled with code deployment and lacked size awareness, causing an `ACCESS EXCLUSIVE` rewrite to block all traffic until completion.
*   **INC-0521**: Zero-priced rate quotes were generated for 22 minutes when a feature flag deletion process left the configuration entry removed but the code reference active, resulting in a default value that triggered a fallback path returning zero rates.

**Common Themes:**

All four incidents stem from **incomplete or flawed operational hygiene**, where critical system states (connection pools, certificate validity, table sizes, and feature flags) were not properly monitored or validated against their actual runtime behavior; they also share the pattern of **reliance on implicit assumptions**—such as "the default is safe," "a missing job cannot fail," or "zero price is valid"—that proved incorrect under specific load or configuration conditions.

It has also picked up on the same common theme, just with a different wording.

Ok, so what about the different reasoning levels - do they make a difference to accuracy, token count and timings?

Marcus decides to run the test looping through all 4 levels and document the difference.

ReasoningLevel	Output tokens	Wall time
none	310	12s
low	6147	73s
medium	6147	75s
high	6147	73s

So thinking level made no difference to the output tokens, output text or wall time.

Qwen’s own CLI tooling treats these as two separate concepts: an effort ladder, and a distinct binary flag for disabling reasoning entirely — with a translation layer that silently clamps unsupported tiers to the nearest supported one. Which matches the measurement: None is a different kind of setting from low/medium/high, and low/medium/high produced byte-identical output. Where the collapse happens — the SDK, Ollama, or the weights — Marcus can’t tell from here.

Refining the thinking

The OpenAI SDK gives us no way to read the thinking. ChatCompletion has no property for it, which is entirely correct behaviour: the SDK models OpenAI’s API, and OpenAI doesn’t return raw reasoning — you get summaries and token counts, not the chain itself.

Ollama does return it. Its OpenAI-compatible layer adds a reasoning field to the message object that the SDK’s response model knows nothing about, so the data arrives and is discarded on the way to the typed surface. GetRawResponse() is the way underneath.

static string? GetReasoning(ClientResult<ChatCompletion> result)
{
    using var doc = JsonDocument.Parse(result.GetRawResponse().Content);
    var message = doc.RootElement.GetProperty("choices")[0].GetProperty("message");

    return
        message.TryGetProperty("reasoning", out var r) ? r.GetString()
        : message.TryGetProperty("reasoning_content", out var rc) ? rc.GetString()
        : null;
}

Two caveats. This only works because Ollama puts the field on the response — point the same helper at OpenAI and it returns null, correctly, because there is nothing there. And it’s parsing a json field by name, with two candidate names because the ecosystem hasn’t agreed on one. Fine for a diagnostic, wrong for anything shipping.

If you’d rather not do that, OllamaSharp and Microsoft.Extensions.AI both surface it on the response directly.

Here’s what it was actually doing:

Reading the model’s thinking process, it seemed to get very confused over a single word in the prompt - “above”. The prompt says “Summarise the incidents above” — and the model has no reliable way to know what “above” means, because the documents arrived as separate user messages. It then spends a long time trying to establish whether there was a previous turn, whether the reports are in this message, whether it already processed INC-0417. “Wait, no” appears about fifteen times. It gets there eventually and correctly, but the cost is large in terms of tokens and time.

Marcus tries a better prompt.

const string UserPrompt =
    "Summarise the incident reports provided in this conversation. There are 4 of them. For each incident, give its identifier, its "
    + "duration, and its root cause in one sentence. Then identify the themes "
    + "common to all of them.";

So this time rather than saying “Summarise the incidents above” - he tells the model that the incidents are in this conversation, and how many of them there are.

He runs that. And pretty quickly realises there is a problem. It runs for a very long time before returning:

----------------------------------------
  DoneReason           : length
  ThinkLevel           : low
  InputTokenCount      : 4732
  OutputTokenCount     : 28036
  TotalTokenCount      : 32768
  ElapsedMilliseconds  : 360351

So this time he had a Done Reason of length

This tells him that the model ran out of context space. The output was so large it filled the whole window. OutputTokenCountof 28036 and Total of 32768 - which is the context size he has set in Ollama.

Marcus couldn’t raise the context any higher — VRAM limits on his workstation. However his next finding would show no context size would of helped. Looking at the thinking traces he sees why:

...
- **Wait.** Okay, let me check if there was a fourth incident report `INC-0521` in the text provided *in this turn*. I will search for "Zero-Priced". It appears at the very end of my context window? No, it ends with INC-0498.
- **Wait.** Okay, let me check if there was a fourth incident report `INC-0521` in the text provided *in this turn*. I will search for "Zero-Priced". It appears at the very end of my context window? No, it ends with INC-0498.
- **Wait.** Okay, let me check if there was a fourth incident report `INC-0521` in the text provided *in this turn*. I will search for "Zero-Priced". It appears at the very end of my context window? No, it ends with INC-0498.
- **Wait.** Okay, let me check if there was a fourth incident report `INC-0521` in the text provided *in this turn*. I will search for "Zero-Priced". It appears at the very end of my context window? No, it ends with INC-0498.
- **Wait.** Okay, let me check if there was a fourth incident report `INC-0521` in the text provided *in this turn*. I will search for "Zero-Priced". It appears at the very end of my context window? No, it ends with INC-0498.
- **Wait.** Okay, let me check if there was a fourth incident report `INC-0521` in the text provided *in this turn*. I will search for "Zero-Priced". It appears at the very end of my context window? No, it ends with INC-0498.
- **Wait.** Okay, let me check if there was a fourth incident report `INC-0521` in the text provided *in this turn*. I will search for "Zero-Priced". It appears at the very end of my context window? No, it ends with INC-0498.
- **Wait.** Okay, let me check if there was a fourth incident report `INC-0521` in the text provided *in this turn*. I will search for "Zero-Priced". It appears at the very end of my context window? No, it ends with INC-0498.
...

The model got stuck in a loop thinking this same line over and over again until it died. By trying to be more precise, Marcus had made it worse!

It was seemingly having a problem finding the 4th incident report, but that report was certainly there as evidenced by the InputTokenCount of 4732, near identical to every other run in this post.

The model couldn’t find INC-0521 while reasoning, and then couldn’t reconcile that with the assertion given by the user, that there were four reports.

Marcus tried the tool again two more times. Exactly the same result.

Maybe the answer is to give the model a concrete rule about where a report starts?

He tries this prompt:

const string UserPrompt =
    "Summarise the incident reports provided in this conversation. "
    + "Each incident report begins with a heading of the form # Incident Report INC-nnnn. Summarise every report present. "
    + "For each incident, give its identifier, its duration, and its root cause, as stated in the report's Root cause section, in one sentence. "
    + "Then identify the themes common to all of them.";

So now rather than telling the model how many reports there are - he tells the model how to detect when a new incident starts.

He runs that:

----------------------------------------
  DoneReason           : stop
  ThinkLevel           : low
  InputTokenCount      : 4761
  OutputTokenCount     : 2971
  TotalTokenCount      : 7732
  ElapsedMilliseconds  : 41716

Ok, that is a lot better. Done reason is now stop so it didn’t overflow this time. And output token count is down to 2971.

And how did it perform in finding common themes?

### Common Themes

All four incidents share three primary themes regarding reliability engineering practices:

1.  **Incomplete Configuration and Code Hygiene:** Changes were made where dependencies or configurations (missing `using` statements, omitted cronjob manifests, flag defaults) were not fully updated together with the codebase.
2.  **Lack of Monitoring on Critical State Changes:** Alerts relied on metrics that did not catch specific failure modes (p50 latency vs saturation, job absence vs presence, price distribution unmonitored), often relying on assumptions about "dead" or irrelevant configurations being safe to ignore.
3.  **Coupling of Deployment and Data/Infrastructure Operations:** Migrations and infrastructure changes were tightly coupled to release pipelines without independent scheduling or inventory checks for large tables, regions, or critical resources.

It still picked up the correct common themes. It worded some of them better. But the best line still comes from the non-thinking run - ‘reliance on implicit assumptions’

Combining all his measurements into one table finds:

Configuration	Output tokens	Wall clock	Result
Thinking off	310	12s	Complete
Thinking on, original prompt	6,147	75s	Complete
Thinking on, count asserted	28,036	360s	length — no answer
Thinking on, pattern given	2,971	42s	Complete

The comparison

For the work Marcus was doing, 310 tokens and 12s vs 2,971 tokens and 42s to achieve the same result; ten times the tokens, three-and-a-half times the wall clock. But was the result better?

Both versions found the same underlying pattern, phrased differently — thinking named the mechanism concretely, None named it more elegantly. Neither is clearly better.

---

Marcus set four different reasoning levels and moved nothing. He changed the wording of one prompt and moved it by a factor of nearly five in the wrong direction, then by a factor of two in the right one.

The parameter he was given to control the cost didn’t. The words he wrote did.