I added a sentence to a passage and the vector did not change

I embedded a passage from the SharePoint export. Then I added one sentence to the end of it and embedded it again. The cosine similarity between the two vectors was 1.0.

Not 0.998. Exactly 1.0.

The sentence had no effect because the model never read it. all-MiniLM-L6-v2 takes 256 word pieces and truncates anything past that, with no error, no warning and nothing in a log. I knew that number before I started, in the way you know a number you have read somewhere. What I had not done was work out what it does to a chunk.

Most of what is written about this model pairs it with chunks of 500 to 1000 tokens. Do that and about half of every chunk is decoration. It sits in the payload, it comes back with the search result, a person reads it and finds it useful, and it played no part in whether that chunk was found in the first place. The first 256 word pieces decide retrieval. The rest is along for the ride.

So how many words is that

Tokens are not words, and the ratio depends on the text. Documentation full of product names and interface labels breaks into more pieces than ordinary prose does, so the general figure people quote was never going to be the right one here.

I measured it on the actual corpus. Over 60 real pages: 13,017 words, 16,126 word pieces, 1.239 pieces per word.

Two special tokens come off the 256, which leaves 254 usable, which is about 205 words at that ratio. The chunk size is 180. That is 205 with headroom for a denser than average page.

Everything else about chunking follows from that one number. Overlap is 30 words, roughly one long sentence, because a sentence that straddles a boundary is otherwise only ever seen as two halves and neither half says what the whole sentence said. Chunks do not span pages, because a page boundary in this export is usually an article boundary, and joining the end of one article to the start of the next gives a vector that is a blend of two subjects and a good match for neither.

There is also a floor of 20 words. Some pages hold nothing but a heading or the word Feedback, and those were becoming chunks of three or four words that embed to something meaningless and can still be returned by a search.

The floor has a cost and I can state it: it removes 52 chunks and takes 52 pages out of the corpus entirely, 1793 down to 1741. Those pages now contribute nothing at all. They also contained nothing worth retrieving, which is the bet I am making.

The comment that was lying

I wrote the chunking tests earlier than the plan said to, because chunking had acquired arithmetic that retrieval was about to depend on, and a boundary that moves by accident shows up weeks later as slightly worse answers with nothing to trace it to.

Writing them meant reading _split properly for the first time. It ran a sentence splitting regex over the page, then flattened every sentence back into one flat list of words and cut on word count. The regex had no effect whatsoever. The docstring said boundaries landed at sentence ends. My own comment said the same thing. They landed wherever the counter ran out.

I proved it before touching anything, with a size of 5 and an overlap of 1:

'alpha beta gamma. delta epsilon'
'epsilon zeta. eta theta iota.'

Cut mid sentence, exactly as counting words would do it. That also explained something I had seen in the real corpus a week earlier and not chased: chunks that begin “1. On your website or team site, select Settings”. A numbered list is not a sentence, and in any case nothing was splitting on sentences.

I did three things about it and kept them apart on purpose. Deleted the dead regex and fixed the comment, because code that claims to do something it does not is worse than code that does nothing. Wrote a test that pins the current behaviour, so that the day sentence aware splitting arrives, that test fails and somebody has to change it deliberately rather than not noticing. Then raised the real fix as its own issue, because it moves every boundary in the corpus and needs a re-ingest and a comparison of retrieval before and after. That is a measurement, not a tidy up.

Twenty tests, 0.19 seconds, no containers. One of them is not about chunking at all. It asserts that the configured chunk size multiplied by 1.24 stays under 254. If somebody raises the size to a number that sounds sensible, embeddings quietly start losing their tails again, and that single line is the only thing in the project that would say so.

What the whole document costs

The export is 1798 pages and 2.24 million characters. Five of those pages have no extractable text, which is what pypdf gives you for a cover page or a full page diagram. They are skipped rather than yielded empty, so the page numbers coming out have gaps in them. A gap in the numbering is visible. A chunk of nothing is not.

python -m ragops.ingest data/sharepoint.pdf does the rest:

The flat rate is the part worth looking at. It says the pipeline does not slow down as the collection fills, which was not guaranteed and would have been the first sign of a design problem.

Nearly all of those 149 seconds is embedding on four cores, and it is only that quick because the embeddings service takes a batch. A hundred chunks one at a time took 26.2 seconds. The same hundred in a single call took 3.5. Scaled up to 2918 chunks that is about thirteen minutes of embedding one at a time against under two batched.

It streams the whole way through. Pages are yielded one at a time, chunks are batched at 128, and each batch is embedded and written to Qdrant before the next page is read, so nothing ever holds the document. The simpler shape, chunk everything then embed everything then upsert, would have worked perfectly well on this corpus. It would also fail on the first document big enough to matter, and it would fail in the embedding step, which is not where the mistake was made. Writing as it goes has a second benefit: a crash at chunk 2000 leaves 2000 chunks searchable.

Which is why the upload endpoint answers early

149 seconds inside a POST means a client holding a connection open for two and a half minutes, a proxy somewhere eventually deciding that is unreasonable, and no way to report progress while it happens.

So POST /documents does the parts that can fail fast, saving the file and opening it with pypdf, and returns 202 with a document id. A text file renamed to .pdf comes back as 400 on the call that sent it rather than as a line in a log ten minutes later. The slow work runs behind, and the id is polled for status.

That status lives in a dict in the API process, so a restart loses every record, and two replicas would each know only about their own uploads. It is honest for what this is today, which is one person uploading a document and waiting a couple of minutes for it. It stops being honest the moment either of those things changes, and the fix is to keep the record in Qdrant next to the chunks.

The number I did not have at the start of any of this was 1.24. Almost every other decision in the pipeline turned out to be downstream of it.