The first real question I put to my locally hosted model came back correct.

A SharePoint site collection is a group of related sites that share common features, settings, and permissions within Microsoft SharePoint.

Then about ninety blank lines.

The response reported 120 completion tokens, which was exactly the max_tokens I had sent. Every request did the same. So it was not finishing and then padding. It was being cut off, and it happened to have said something useful before it ran out.

The theory that fitted

Phi models emit <|end|> when a turn is over. If the server is not treating that as a stop token the model keeps going, which is precisely what I was looking at.

My model config had this:

parameters:
  model: phi-3.5-mini-instruct-Q4_K_M.gguf
  stopwords:
    - "<|end|>"

LocalAI documents stopwords at the top level of the config, not nested under parameters. So it was being silently ignored, the model had no stop tokens, and I had solved it before finishing my tea.

I moved it up a level, restarted the container, waited out the two and a half minutes it takes to load 2.2GB of weights, and asked the same question.

120 tokens. Answer. Ninety blank lines.

The fix that made it stranger

Right, so send the stops on the request instead of trusting the config.

Before doing that I wanted to know whether request level stops worked at all, so I sent one I knew would fire immediately, two newlines. The reply came back empty after two tokens. Useful in two ways: the mechanism worked, and the model was apparently opening its reply with two newlines.

With the real stop sequences in place the blank lines went away. This arrived instead:

A SharePoint site collection is a group of related sites… What are the main components of a SharePoint site collection? The main components include… How can you manage permissions within a site collection? Permissions can be managed through…

It answered me, invented a follow up question, answered that, invented another, and carried on until it hit the limit.

Two fixes in, and the output was weirder than when I started.

The number I had already been given

The response body carries a usage block. Mine said this:

"usage": {"prompt_tokens": 12, "completion_tokens": 120, "total_tokens": 132}

Twelve prompt tokens. For a system message describing the assistant’s job, plus a question about SharePoint site collections.

That is not enough for either of them, never mind both.

So the messages I was sending were not arriving as messages. No <|system|>, no <|user|>, no <|assistant|>. LocalAI had no chat template configured for this model and was handing llama.cpp something close to raw text.

Which explains the behaviour completely. There was no conversation to end. There was a piece of text, and the model’s job is to continue text. So it continued it: first with a plausible next paragraph, then with a plausible next question, because a document containing a question and an answer usually contains another question after that.

It was never failing to stop. Nothing had told it there was anywhere to stop.

The actual fix

template:
  chat_message: |
    <|{{.RoleName}}|>
    {{.Content}}<|end|>
  chat: |
    {{.Input}}
    <|assistant|>

One restart, same question:

A SharePoint site collection is a grouping of SharePoint sites that share common features, permissions, and administration settings within a single site collection.

One sentence. Nothing after it.

What I got wrong

Not the diagnosis. The order I read the evidence in.

Everything needed to identify this was in the first response I ever received. completion_tokens matching max_tokens said it was being truncated rather than finishing. prompt_tokens of 12 said the prompt was not what I thought it was. I read the first number, built a theory that fitted it, and did not look properly at the second until two failed fixes later.

I kept the request level stop sequences anyway, even though the template fixed the cause. They are <|end|>, <|user|> and <|system|>, none of which turn up in an answer about SharePoint, so the chance of them cutting off something real is close to zero. If the template is ever lost in a config change, the failure becomes a slightly odd answer instead of a runaway one.

The general version, which I will be reusing: when a model behaves strangely, the useful question is usually not what the model did. It is what the model was actually sent. Those two are separated by a template you probably did not write, and the token counts in the response are the cheapest way to notice you have been looking at the wrong one.