Most explanations of retrieval augmented generation stop at the diagram. Question goes in, arrow to a vector database, arrow to a model, answer comes out. It is accurate and it teaches you nothing, because every interesting decision is inside the arrows.
I have just built the retrieval half of one. Not the whole pipeline, ingestion is next, but text can now be stored and searched back, and getting there meant making choices that nobody explains in the diagram. Those choices, and what happened when I finally tested them, are the subject.
The problem keyword search has
Someone asks:
Why can this person still see the file after I removed them from the group?
The documentation that answers this almost certainly does not contain the phrase “still see the file”. It talks about permission inheritance, unique permissions, and direct assignments. Keyword search needs the asker to already know the words the document uses, which means it works best for the people who need it least.
What you want is search by meaning. Which requires meaning to be something a computer can compare, and that is the entire idea behind embeddings.
384 numbers
An embedding model reads a piece of text and produces a list of numbers. The model I am using, all-MiniLM-L6-v2, produces 384 of them. Every passage I store becomes 384 numbers, and every question becomes 384 numbers in the same space.
The numbers are not readable. Nobody looks at position 172 and says ah, permissions. They are learned during the model’s training, and what they encode is only meaningful relative to each other. Two passages about permission inheritance end up with similar numbers. A passage about retention policies ends up somewhere else.
Search then becomes geometry. Take the question’s 384 numbers, find the stored passages whose 384 numbers point in a similar direction, return those.
That is the whole trick. It is why “still see the file” can find a page that never uses the phrase, and why the demo looks like magic until you find the case where it retrieves something confidently irrelevant.
Similar in which sense
Two vectors can be close in two different ways, and you have to pick.
Cosine similarity compares direction only. Dot product compares direction and magnitude together, so a longer vector scores higher regardless of what it means.
My embeddings come back normalised, meaning every one has a length of exactly one. I checked, and the magnitude of a returned vector is 1.0. With unit vectors, cosine and dot product rank identically, so today the choice changes nothing.
I configured the collection with cosine anyway. If I ever swap the embedding model for one that does not normalise its output, dot product would quietly start ranking longer vectors higher, nothing would look broken, and the results would just be slightly wrong in a way that is very hard to notice. Cosine says what I mean.
One collection, not one per document
Qdrant, the vector database I am using, organises vectors into collections. The obvious design is a collection per document: deleting a document is a single drop, and searching one document needs no filtering at all.
It falls apart as soon as somebody asks a question that spans documents. You would have to query every collection, then merge and re-rank the scores yourself, which is precisely the work you brought a vector database in to do. Qdrant also maintains an index per collection, so a few hundred documents means a few hundred indexes.
So: one collection, with the document id attached to each vector as payload, and a keyword index on that field. Searching everything is the default. Searching one document is a filter. Deleting a document is a delete by filter rather than a drop, which is slower, and it is the price for everything else being simpler.
The index on that field goes in when the collection is created rather than later. Without it, a filtered search scans everything. That is fine with ten documents and not fine with a thousand, and adding an index to a collection that already holds data is a considerably less pleasant afternoon.
The part that surprised me
Embedding is not free, and it is not linear in the way you expect.
I timed it. Embedding 100 passages one at a time took 26.2 seconds. Sending all 100 in a single batched call took 3.5 seconds. Same model, same machine, same text, seven and a half times faster.
The saved HTTP round trips are not the reason. The model processes 32 texts per forward pass, so a loop of single calls does one text’s worth of work in a slot that could have held 32. That difference is what decides whether indexing a document set is an afternoon or a week, and it is why the service has a batch endpoint that returns a list rather than an endpoint you are expected to call in a loop.
What it actually retrieved
None of the above is worth anything as a claim, so I tested it. Six SharePoint admin passages stored, then four questions worded to avoid the vocabulary of the passages, because matching on shared words is what keyword search already does.
The correct passage came first for two of the four, and was in the top three for all four.
The two it did not get first are the interesting ones.
“Why can this person still see the file after I removed them from the group” returned the passage about sharing links, scoring 0.241, with permission inheritance third. Nothing scored well. The passage that answers that question talks about inheritance and unique permissions, and the question talks about groups and files, so there is less overlap in meaning than I assumed when I wrote it.
“How do I stop staff sending documents to people outside the company” returned sharing links first at 0.366, with external sharing second. I had marked that wrong. Looking at it again, it is arguably the better answer, because sharing links are the mechanism by which documents leave an organisation. The retrieval was fine. My expected answer was the debatable part.
That second one changed how I plan to measure this properly. Grading against a single correct passage per question would have counted a good answer as a failure. The real evaluation set needs a set of acceptable passages per question, which is more work to build and the only version that means anything.
Six passages is far too small to conclude anything, and I am not going to tune against four questions I wrote myself. It is a baseline, and the point of a baseline is that the next number can be compared to it.
What is still missing
Everything above stores and finds passages that I wrote by hand. Turning real documents into those passages is next, and it is where the decisions get harder: how big a chunk should be, how much neighbouring chunks should overlap, and what to do about a PDF page that returns no extractable text because it is a scan.
Chunk size in particular is a proper trade. Too small and a passage loses the context that made it meaningful. Too large and the useful sentence is diluted by everything around it, and it eats the model context window when several are retrieved at once.
I do not know my answer to that yet. When I have one it will be because I measured it against questions with known correct sources, rather than because a tutorial suggested a number.