RAG has changed the way AI models work with corporate data. Thanks to this approach, there is no longer any need to constantly retrain a model; it is enough to connect it to information sources (databases, APIs, internal documents) so that it generates answers in real time and with real context. But behind the success of any RAG system there is a component that tends to be underestimated: how that information is fragmented and organised before it is indexed.
Good chunking design makes the difference between an assistant that answers ambiguously and one that delivers accurate, verifiable and fast answers.
How chunking works step by step
Each chunk is converted into an embedding, that is, a numerical vector that represents its meaning. When the AI receives a query, it does not compare it against the whole document, but against the embeddings of the chunks already indexed. The system retrieves the most relevant fragments and uses them as context to generate the answer. How it is decided where to cut the document is precisely what distinguishes one chunking strategy from another.
Chunking strategies: which to choose for each use case
There is no single way to fragment a document, and choosing well depends on the type of content and on what the system is going to be asked.
Fixed-size chunking cuts the text every certain number of tokens, without regard to meaning. It is the cheapest to compute and, according to recent benchmarks, remains a solid starting point in most cases, especially in its recursive variant, which first respects paragraphs and sentences before forcing a cut (Firecrawl, 2026).
Semantic chunking measures the similarity of meaning between consecutive sentences and cuts where the topic changes. It usually improves recall in documents with clear thematic transitions, although it requires generating one embedding per sentence, which makes the process more expensive.
Hierarchical chunking indexes small fragments for search, but retrieves the larger parent fragment to give the model as context. It is useful when precision in search is needed without losing context in the answer.
Agentic chunking delegates the decision of where to cut to a language model, identifying complete units of information (a clause, a step in a procedure). It offers the best retrieval quality in highly structured documents, such as contracts or regulations, but its indexing cost can be much higher than that of fixed-size chunking.
Late chunking reverses the usual order: embeddings are first generated over the complete document and it is split afterwards, so that each fragment retains part of the global context of the document it comes from.
The general recommendation is not to choose the most sophisticated technique, but to start with the simplest and cheapest one, and only increase complexity when the retrieval metrics in your own environment justify it.
Chunk size: 200, 500 or 1,000 tokens?
Chunk size is one of the most decisive factors in the performance of a RAG system. Let us look at the differences:
| Chunk size | Advantages | Disadvantages | Recommended uses |
|---|---|---|---|
| 200 tokens | High precision in very specific answers. Low computational cost. | Risk of losing context. Greater number of queries to the index. | FAQs, direct answers, short support snippets. |
| 500 tokens | Good balance between context and precision. Efficient retrieval. | May mix information if documents are not well structured. | Technical documentation, internal manuals, legal bases. |
| 1,000 tokens | Retains broader context, useful for long explanations. Less fragmentation. | Higher latency and risk of irrelevant noise. Higher indexing cost. | Complex reports, long articles, academic documents. |
The general rule is to match the chunk size to the type of information and the use case.
Precision and recall: the balance that must be measured
Information retrieval systems measure two different things: precision, the percentage of retrieved fragments that are actually relevant, and recall, the percentage of the available relevant information that is retrieved. Small chunks tend to maximise precision, because each fragment is very specific, but they sacrifice recall, since related information may end up in another fragment. Large chunks do the opposite: they improve recall by including more content per fragment, but lose precision because the model may incorporate irrelevant noise into the answer. Medium-sized chunks, around 500 tokens, usually offer the best balance and are the most common standard in corporate RAG implementations.
To know whether a particular pipeline is well tuned, intuition is not enough: it is advisable to measure it with an evaluation framework such as RAGAS, which makes it possible to calculate context precision and recall without relying on manual annotations, comparing different chunk sizes against the same real questions.
Latency and computational efficiency
Chunking affects not only the quality of the answers but also the performance of the system. Small chunks require more embeddings and more comparisons, which increases latency in searches over large document volumes. Large chunks reduce the number of embeddings and queries, which lowers retrieval latency, but raises the cost of generating the initial embeddings. In environments with documents that change frequently, small chunks are usually more manageable because they allow only the modified part to be re-indexed; in static repositories, large chunks reduce indexing costs.
Contextual retrieval: an additional layer on top of chunking
One of the most significant advances of recent years does not replace chunking, but complements it. In 2024 Anthropic proposed contextual retrieval, a technique that adds to each chunk a brief explanation of its role within the complete document before converting it into an embedding, so that the fragment no longer depends on the rest of the text to make sense. According to the data published by Anthropic, combining contextual embeddings with reranking can reduce retrieval failures by up to 67 per cent in the top twenty retrieved fragments (Anthropic, 2024). It is a particularly useful technique when several documents deal with similar concepts in different contexts, something common in corporate knowledge bases.
Chunking best practices
Respect the natural structure of the document, avoiding cutting sentences in half and using paragraphs, sections or headings as a guide. Applying an overlap of between 50 and 100 tokens between chunks helps avoid loss of context at the cuts, although it is worth validating this in your own environment, as some recent analysis has found cases where overlap brings no clear improvement and does increase indexing cost. Enriching each chunk with metadata (author, date, topic or source) improves filtering and the traceability of answers. And, above all, it is advisable to evaluate with real metrics, testing different chunk sizes and strategies and measuring precision, recall and latency on questions representative of your own use case, instead of assuming that a standard size will work equally well on any corpus.
Applied example: a 500-page technical manual
A 500-page corporate technical manual illustrates these differences well. With 200-token chunks, the system answers very specific questions with great precision, such as the voltage limit of a particular piece of equipment. With 500-token chunks, the result is better suited to step-by-step instructions, retaining enough context to explain how to configure a module. With 1,000-token chunks, the system performs better on broad explanations or internal regulations, where the answer needs more global context.
Chunking and regulatory traceability
An additional benefit of well-designed chunking is explainability: the assistant can show not only the answer, but the exact fragment of the document from which it was extracted. This is not just a question of user trust. Regulation (EU) 2024/1689 on artificial intelligence requires high-risk AI systems to have technical capabilities for the automatic recording of events throughout their life cycle, precisely to ensure a level of traceability appropriate to their purpose (Regulation (EU) 2024/1689, EUR-Lex). Chunking that allows each answer to be traced back to its source fragment makes it easier to meet that requirement, although it is worth checking case by case whether the specific system falls into the Regulation's high-risk category.
How we approach it at LurNova
At LurNova we design RAG pipelines that intelligently adjust chunking according to the type of document, the use case and the level of precision required, as part of our RAG solutions. Our work combines adaptive chunk sizes according to the content, overlap and enriched metadata to maximise context without losing precision, and integration with MCP so that the AI can access data sources securely and efficiently. If your organisation is assessing how to structure its knowledge base for AI, in our AI consultancy we analyse the specific case before recommending an architecture.
Because in AI it is not enough to have the information: it must be structured intelligently so that it becomes useful, actionable knowledge.
Co-founder and CTO of LurNova, with more than 20 years' experience in developing digital technologies. Specialised in designing secure, private and governed artificial intelligence architectures and solutions for organisations.
