Every organisation I have worked with has the same pile of documents. Procedures, policies, standards, incident write ups, half of it in SharePoint and the other half in someone’s OneDrive. People cannot find anything in it, and a question that should take thirty seconds turns into asking three colleagues.

That is the problem retrieval augmented generation is genuinely good at. It is also the problem where the first question from anyone senior is not about accuracy. It is: where does the document go when someone asks a question about it?

For a lot of places, particularly government and resources, that question ends the conversation. Not because the answer is definitely no, but because nobody wants to be the person who authorised sending internal procedures to a hosted model without a data handling decision behind it.

So I built the version where the answer is easy. Nothing leaves the machine.

What that means in practice

The system I have been building runs entirely on my laptop. Phi-3.5-mini as the language model, served through LocalAI. all-MiniLM-L6-v2 behind a small FastAPI service to turn text into vectors. Qdrant to store and search those vectors. Docker Compose to run the lot.

The question, the passages retrieved from your documents, the prompt assembled from them, and the answer all stay on the hardware you control. There is no API key because there is nothing to authenticate to.

The machine is an i7-1165G7 with four physical cores. Deliberately unremarkable. If it runs there it will run on anything an organisation already owns.

The honest caveats, before the numbers

It needs the internet once. The model weights are 2.2GB and have to come from somewhere. After that, inference is local and can run disconnected.

Local is not secure. Not sending documents to a hosted model removes one path of exposure. It does nothing about logs, open ports, file permissions, backups, or a retrieval layer that cheerfully returns a passage the person asking is not supposed to see. That last one is the interesting problem in an enterprise deployment, and it is not solved by where the model runs.

Small model, narrow job. Phi-3.5-mini is 3.8 billion parameters, quantised to about 2.2GB. It is not going to reason its way through a novel problem. Its job here is reading three or four retrieved passages and explaining what they say, which it is fine at.

What it costs, measured

Answering a question: 8 to 17 seconds, warm.

Cold start: about 160 seconds after a container restart, which is the model being read into memory. This one caught me out. My original health check went green in ten seconds because it asked the server whether it knew about the model, not whether the model could answer. Anything depending on it would have started against a model that was still loading and timed out.

Indexing: embedding 100 chunks of text one at a time took 26.2 seconds. Batched into one call, 3.5 seconds. That difference is the whole reason a batch endpoint exists, and it is what determines whether indexing a large document set takes an afternoon or a week.

Concurrency: one person at a time. Two questions at once compete for the same four cores. This is the number that decides whether the approach fits, and it is the one most demos never mention.

Money: no per question cost. The cost moves to electricity, CPU time, disk, and the hours spent keeping it running. For a single user experimenting, that is a good trade, because I can run the same forty questions fifty times while changing how documents are split up, and the bill does not move.

Where I would stop recommending it

A laptop is a proof, not a deployment. If a team of thirty is going to use this, the four core answer stops working immediately, and the honest options are a server with a GPU on infrastructure the organisation controls, or a managed model service with a data handling agreement that someone has actually read.

The interesting part is that the decision is no longer ideological. Once you have measured 8 to 17 seconds for one user on hardware you already own, you can work out what thirty concurrent users needs, and you can compare that against the compliance cost of the alternative. That is a conversation with numbers in it rather than a conversation about whether AI is safe.

What does not change is the architecture. Retrieval, embeddings, a vector store, prompt assembly, and evaluation are the same whether the model runs on a laptop or behind an API. Only the last box changes.

The part I care about most

None of the above says whether the answers are any good.

That is decided almost entirely by retrieval. If the right passage is not found, no model rescues the answer, and a larger one just produces a more confident wrong one. Every document chatbot demo I have seen skips this, because measuring it is work and demoing three good answers is not.

The corpus I am using is SharePoint and Microsoft 365 administration documentation, and that is not an accident. I spent about ten years in that stack. When the system answers a question about the difference between a site owner and a site collection administrator, I can tell immediately whether it is right. Most people building these cannot say that about their own corpus, which is exactly why so few of them are evaluated.

The next piece of work is a set of around forty of those questions with the source sections that should answer them, scored on whether retrieval actually finds them. I will publish those numbers, including the questions it does badly on. Those are the interesting ones.