Thursday, September 17, 2026

Chunking strategy in document embedding generation:

• One vector per document = is this document, as a whole, similar to the query?
  • Dilutes single relevant paragraph
• One vector per chunk = does the document contain something similar?

In all-MiniLM-L6-v2 (hey, that's what I use!) model, 512 tokens = ~380 English words

Four strategies of note:

• truncate - what I've been using so far
  • If most documents fit within the token window = keep
• fixed - chunk by the token window size
  • Loses context when a thought is truncated in the middle = poor embedding
  • Use when there is no reliable the structure
• recursive - a chunk stops where the text stops, not where the counter runs out
• sentence - use when a fragment of a sentence changes or destroys the meaning
  • e.g. support tickets, email threads, chat/meeting transcripts, legal/policy text, news, customer reviews, medical/scientific abstracts
  • Fewer, cleaner chunks

Chunk size is the setting that actually affects your results. The trade is direct: a smaller chunk is a sharper match on one idea, a larger chunk carries more context but dilutes each idea inside it. A paragraph buried in a long document only becomes findable once the chunk size is small enough to give it a vector of its own.