I restarted LocalAI and Docker called it healthy after about ten seconds. Then I asked the model a question and sat there for nearly two minutes waiting for the answer.

Nothing had crashed. The container was up, the port was open, the health check was green, and the thing still could not do its job.

What the check was actually asking

I am building a local RAG system over SharePoint and Microsoft 365 admin documentation. It runs on a laptop, on CPU, with no API keys. The language model is Phi-3.5-mini quantised to Q4_K_M, served by LocalAI, and its weight file is 2.2GB.

My health check was this:

GET /v1/models

Which looked completely reasonable. LocalAI lists the models it knows about, my model was in the list, so the model must be ready.

It is not the same statement. /v1/models reads the config LocalAI parsed at startup. It answers as soon as the HTTP server is listening, and the HTTP server is listening long before 2.2GB of weights have been read off disk and arranged in memory. I was asking “do you know this model exists” and reading the answer as “can you use it”.

Measuring instead of guessing

I could have left it there. Nothing depended on LocalAI yet, so the only symptom was one slow question, and one slow question on a laptop running a 3.8B model is not surprising enough to investigate.

What made me look was that the slowness was not consistent. The first question after a restart took two minutes. The next took eight seconds. If the model were simply slow, both would be slow.

So I forced a recreate and watched the status directly:

docker inspect --format '{{.State.Health.Status}}' ragops-localai-1

Every twenty seconds. It said starting for 140 seconds, then healthy at 160. Except that was with the fixed check already in place. With the original one it went green at about ten seconds and the model was not usable for another 150.

That gap is the whole bug. There is nothing wrong with Docker and nothing wrong with LocalAI. The endpoint did exactly what it documents. I had written a check that tested the wrong thing and then trusted it.

Why it matters before it breaks anything

At this point in the project nothing else in compose depends on LocalAI, so the misleading status cost me one slow request and no more.

That changes the moment the API arrives. Compose can wait for a dependency to be healthy before starting the thing that uses it, which is the point of having health checks at all. With the old one the sequence would be: LocalAI opens its port, compose marks it healthy, the API starts, the first question arrives, and the request times out against a model that is still loading.

The error would surface in the API. The cause would be in a service that reported itself healthy. That is the expensive kind of bug, the one where the component showing the symptom is not the component at fault, and you spend an evening reading the wrong logs.

The fix

Ask it to do the smallest real version of its job:

curl -sf -m 120 http://localhost:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"phi-3.5-mini","messages":[{"role":"user","content":"ok"}],"max_tokens":1}' \
  | grep -q choices

One token. I do not care what it says, only that the whole path worked: request accepted, model found, weights in memory, inference engine ran, completion returned.

The trade is that every check is now a real inference, so the interval went from 30 seconds to 5 minutes and the timeout to 130 seconds. Once the model is loaded that call takes under a second, so the ongoing cost is small. The start period is 45 minutes, which sounds absurd until you remember the first run downloads 2.2GB before it can start loading anything.

The same question, a different answer

The embeddings service in the same stack runs all-MiniLM-L6-v2 behind FastAPI. Same problem in principle: the port opens before the model is usable.

I did not give it the same fix. Its /health returns 503 until a lifespan handler has loaded the model and embedded one string, and the compose check just reads /health. MiniLM loads in 4.7 seconds and the container is healthy about 21 seconds after start, so running an embedding every 30 seconds forever would buy nothing that the startup check has not already proved.

That is the part I would want to be asked about. The principle is the same, the answer is different, and the difference is the load time. A check that costs a real inference is worth it when the alternative is a two and a half minute lie. It is not worth it when the service is ready in five seconds.

I also added a small script, scripts/wait-for-stack.sh, that polls each service until it can actually do its job. Compose can only express dependencies between services it starts. I needed the same answer from the terminal before running anything by hand.

If this were Kubernetes

Compose gives you one health check and a start_period. Kubernetes splits it into three, and this is exactly the case the split exists for.

A startup probe with a generous failure threshold, so a model that takes two and a half minutes to load is not mistaken for a crash loop. A readiness probe that proves it can serve, which takes the pod out of the service endpoints without killing it. A liveness probe that is cheap and only catches a genuinely stuck process. While the startup probe is still running, the liveness probe is not allowed to restart the pod, which is the property that makes slow loading models workable at all.

Doing it in compose is the same reasoning with cruder tools.

What I took from it

I had treated two sentences as the same thing: the server knows about the model, and the model can answer. A health check is only worth having if it proves the smallest version of the work the user actually needs.