Crafting Coherent Context: Moving Beyond Simple Chunking with Advanced Context Assembly

Introduction: The Quest for Perfect Context

Welcome back, fellow RAG adventurers! In our previous chapters, we laid the groundwork for Retrieval-Augmented Generation (RAG) by understanding its core components and the importance of effective retrieval. We briefly touched upon how breaking down documents into smaller pieces, or “chunks,” is crucial for feeding relevant information to our Large Language Models (LLMs).

But here’s a little secret: while simple chunking is a good starting point, it’s often the Achilles’ heel of basic RAG systems. Why? Because the way we prepare and present context to our LLM profoundly impacts the quality, accuracy, and relevance of its generated answers. If the context is fragmented, incomplete, or distorted, even the smartest LLM will struggle to provide a truly insightful response.

In this chapter, we’re going to level up our RAG game by diving into advanced context assembly techniques. We’ll explore how to move beyond the limitations of simple fixed-size chunks to create richer, more coherent, and query-aware contexts. Get ready to transform your RAG system from good to great by ensuring your LLM always has the perfect information at its fingertips!

The Problem with Simple Chunking: When Less Isn’t More

Before we explore solutions, let’s truly understand the problem. What exactly is “simple chunking,” and why does it fall short for complex RAG 2.0 scenarios?

What is Simple Chunking?

Imagine you have a long document, say, an entire book chapter. Simple chunking involves splitting this chapter into smaller, manageable pieces, usually of a fixed character or token count (e.g., 500 tokens), often with some overlap between chunks to maintain continuity. Each of these chunks is then embedded into a vector and stored in your vector database.

When a user asks a question, your RAG system:

1. Embeds the query.
2. Finds the most similar vector chunks in the database.
3. Retrieves these top-K chunks.
4. Feeds them directly to the LLM along with the user’s query.

Seems straightforward, right? And for many simple questions, it works!

The Silent Killer: Context Distortion

The real challenge arises with more complex queries, especially those requiring nuanced understanding or information spread across different parts of the original document. Simple chunking often leads to context distortion, where:

• Key information is split: A critical sentence might be at the end of one chunk and its supporting detail at the beginning of the next, leading to fragmented context.
• Irrelevant information is included: A chunk might contain the answer but also a lot of surrounding noise, diluting the signal for the LLM.
• Lost global context: By focusing on small, isolated chunks, the LLM might miss the broader narrative or relationships between entities that span an entire paragraph or section.

Let’s visualize this:

In the diagram, if only Chunk 3 is retrieved, the LLM might not know what “concept X” is, or its application, leading to a poor explanation of its limitations. If all three are retrieved, the LLM still needs to piece them together, which isn’t always efficient.

This is where RAG 2.0 steps in, offering sophisticated methods to craft a truly coherent context.

Advanced Context Assembly Techniques

The goal of advanced context assembly is to provide the LLM with just enough relevant information, structured in a way that minimizes cognitive load and maximizes understanding. We want to be precise in retrieval but comprehensive in context.

1. Sentence Window Retrieval

Imagine you’re looking for a specific sentence in a book. Once you find it, you usually read the surrounding sentences or even the whole paragraph to understand its full meaning, right? That’s precisely what Sentence Window Retrieval does.

How it works:

1. Small Chunks for Retrieval: Instead of embedding large chunks, we embed individual sentences (or very small, fine-grained units) into our vector database. These small units are excellent for precise semantic search.
2. Larger Context for LLM: When a query matches one or more of these small “window” chunks, we retrieve not just the matching sentence, but also its surrounding sentences from the original document. This expanded context (the “window”) is then given to the LLM.

Why it’s powerful:

• Precision: Searching at the sentence level allows for highly accurate semantic matching.
• Rich Context: The LLM receives the precise match plus its immediate, natural context, which is often crucial for interpretation.
• Reduced Noise: By retrieving small chunks and only expanding relevant ones, we avoid feeding the LLM large, noisy blocks of text.

2. Auto-Merging Retrieval (or Parent Document Retrieval)

This technique takes the idea of “retrieve small, provide large” a step further. It’s particularly useful when an answer might be spread across several related small chunks that, when combined, form a more complete picture.

How it works:

1. Hierarchical Chunking: You create two sets of chunks from your documents:
   • Small, granular chunks: These are embedded and stored in the vector database for retrieval (e.g., individual sentences or small paragraphs).
   • Larger “parent” chunks: These are the original, larger blocks of text from which the small chunks were derived (e.g., full paragraphs, sections, or even entire documents). These parent chunks are not directly embedded for retrieval but are stored as reference.
2. Intelligent Merging: When your query retrieves multiple small chunks that are all derived from the same parent chunk, the system intelligently “merges” them by retrieving the full parent chunk instead of just the individual small ones. If only one small chunk is retrieved, it might still provide its parent for broader context.

Why it’s powerful:

• Cohesion: Ensures the LLM receives a naturally coherent block of text, even if the individual matching pieces were small.
• Flexibility: Balances the precision of small chunks with the completeness of larger contexts.
• Contextual Depth: Helps resolve situations where an answer requires understanding the broader context of several related sentences.

3. Hierarchical Chunking

This strategy is about offering choices. Instead of one-size-fits-all chunks, you create chunks at multiple levels of granularity.

How it works:

1. Multi-level Chunks: Divide your document into:
   • Level 1 (Coarse): Full sections or entire documents.
   • Level 2 (Medium): Paragraphs or sub-sections.
   • Level 3 (Fine): Individual sentences or very small paragraphs.
2. Adaptive Retrieval: Based on the query’s complexity or the initial retrieval results, your system can decide which level of chunk to retrieve.
   • A broad query might benefit from a high-level summary chunk.
   • A specific question might need a fine-grained sentence.
   • An LLM agent (which we’ll cover later!) could even decide which level to query dynamically.

Why it’s powerful:

• Optimized Context: Provides the most appropriate level of detail for any given query.
• Efficiency: Avoids overwhelming the LLM with too much detail for high-level questions, and ensures enough detail for specific ones.

4. Summary-Based Chunking (Abstractive Summarization)

Sometimes, the original text is simply too dense, or you need a very high-level overview before diving into details. This is where LLMs themselves can help in context preparation.

How it works:

1. Generate Summaries: For each large document or section, use an LLM to generate a concise, abstractive summary.
2. Embed Summaries: These summaries are then embedded and stored in your vector database.
3. Retrieve and Refine:
   • For initial queries, retrieve the relevant summaries.
   • If the LLM or user needs more detail, use the retrieved summaries to identify the original full documents/sections, and then retrieve those.

Why it’s powerful:

• Reduced Noise: Summaries filter out irrelevant details, focusing on core concepts.
• Concise Context: LLMs can process summaries much faster, leading to quicker responses.
• Multi-Stage Retrieval: Enables a “drill-down” approach, starting broad and getting more specific.

5. LLM-Assisted Context Structuring

This is the cutting edge! Instead of just retrieving chunks, we use an LLM before the final generation step to process and restructure the raw retrieved information into a perfectly tailored context.

How it works:

1. Initial Retrieval: Retrieve raw chunks using any of the above methods.
2. LLM as Context Curator: Pass these raw chunks, along with the user’s query, to another LLM (or the same one with a specific prompt). This LLM’s job is to:
   • Identify key entities and relationships.
   • Synthesize information from disparate chunks.
   • Rephrase or reorder the retrieved text to form a coherent narrative.
   • Remove redundant or irrelevant sentences.
   • Create a structured answer outline.
3. Final Generation: The refined, structured context is then passed to the final LLM (often the same one) for generating the ultimate answer.

Why it’s powerful:

• Hyper-Relevant Context: The LLM receives context that’s not just retrieved, but actively curated and optimized for the specific query.
• Addresses Fragmentation: Can bridge gaps between fragmented chunks by synthesizing information.
• Complex Reasoning: Enables the RAG system to handle queries requiring deeper understanding and synthesis across multiple sources.

Step-by-Step Implementation: Sentence Window Retrieval with LlamaIndex

Let’s get our hands dirty and implement Sentence Window Retrieval using Python and the llama-index library. llama-index (version 0.11.x as of 2026-03-20) provides excellent abstractions for these advanced RAG patterns. We’ll use Python 3.11.

First, ensure you have llama-index installed.

pip install llama-index==0.11.10 openai

Note: llama-index often uses OpenAI’s models by default. You’ll need to set up your OPENAI_API_KEY environment variable for the LLM and embedding models. If you prefer a local or open-source model, llama-index supports many options, but for simplicity, we’ll assume OpenAI for this example.

# Set your OpenAI API Key as an environment variable
# import os
# os.environ["OPENAI_API_KEY"] = "YOUR_OPENAI_API_KEY"

# Or set it directly in the code (less secure for production)
# from openai import OpenAI
# client = OpenAI(api_key="YOUR_OPENAI_API_KEY")

Now, let’s write the code:

1. Prepare Your Data

First, we need some text to work with. Let’s create a simple document.

# filename: sentence_window_rag.py

from llama_index.core import Document, VectorStoreIndex
from llama_index.core.node_parser import SentenceWindowNodeParser
from llama_index.core.postprocessor import MetadataReplacementPostProcessor, SentenceTransformerRerank
from llama_index.llms.openai import OpenAI
import os

# Set your OpenAI API Key. Replace with your actual key or set as environment variable.
# os.environ["OPENAI_API_KEY"] = "sk-..." # Recommended to set via environment variable
# If you don't have an OpenAI key, you can use a local LLM or a free tier one.
# For simplicity, we assume OpenAI is configured.

# --- 1. Define our document ---
document_text = """
The Amazon rainforest is the largest rainforest in the world, spanning across nine countries. It is home to an incredible diversity of flora and fauna, including many species not found anywhere else on Earth. Deforestation in the Amazon is a critical environmental concern, primarily driven by cattle ranching and agriculture. Protecting this vital ecosystem is crucial for global climate stability and biodiversity. Recent studies indicate that the rate of deforestation has slightly decreased in some areas due to increased conservation efforts. However, challenges remain, such as illegal mining and logging.
"""

documents = [Document(text=document_text)]

print("Original Document:")
print(document_text)
print("-" * 50)

2. Configure Sentence Window Retrieval

Here’s where the magic happens. We’ll use SentenceWindowNodeParser to create our fine-grained sentence chunks for embedding. Then, MetadataReplacementPostProcessor will ensure that when a sentence is retrieved, we replace its content with the full “window” from the original document.

# filename: sentence_window_rag.py (continued)

# --- 2. Configure Sentence Window Retrieval ---
# Create a SentenceWindowNodeParser
# `window_size` determines how many sentences before and after the retrieved sentence to include.
node_parser = SentenceWindowNodeParser.from_defaults(
    window_size=3, # Include 3 sentences before and 3 sentences after the retrieved sentence
    sentence_splitter=lambda text: text.split(".") # Simple split by period for demo
)

# Parse the document into nodes (sentences)
nodes = node_parser.get_nodes_from_documents(documents)

# Create a VectorStoreIndex from these nodes
# This will embed each sentence and store it.
index = VectorStoreIndex(nodes)

print(f"Number of nodes (sentences) created: {len(nodes)}")
print(f"Example node text for embedding: '{nodes[0].text}'")
print("-" * 50)

Explanation:

• SentenceWindowNodeParser: This parser specifically designed for sentence window retrieval.
• window_size=3: This means if a sentence is retrieved, we will fetch 3 sentences before it and 3 sentences after it from the original document to form the “window” context for the LLM.
• sentence_splitter: A simple lambda to split our text into sentences. For real-world applications, you’d use a more robust sentence tokenizer (e.g., from NLTK or SpaCy).
• VectorStoreIndex(nodes): This creates an index where each of our parsed sentences (nodes) is embedded and stored.

3. Perform Retrieval and Observe Context

Now, let’s query our index and see the difference in the context provided to the LLM. We’ll simulate the LLM’s input.

# filename: sentence_window_rag.py (continued)

# --- 3. Perform Retrieval and Observe Context ---
query_engine = index.as_query_engine(
    similarity_top_k=2, # Retrieve top 2 sentences
    # The postprocessor is crucial for replacing the retrieved sentence with its window
    node_postprocessors=[
        MetadataReplacementPostProcessor(target_metadata_key="window"),
        # Optional: Add a reranker for better quality
        # SentenceTransformerRerank(top_n=2, model="BAAI/bge-reranker-base")
    ]
)

user_query = "What are the main causes of deforestation in the Amazon?"

print(f"User Query: '{user_query}'")
print("-" * 50)

# Get the response
response = query_engine.query(user_query)

# We can inspect the source nodes to see the actual context provided to the LLM
print("Context provided to LLM (expanded window):")
for i, node in enumerate(response.source_nodes):
    print(f"\n--- Retrieved Node {i+1} ---")
    print(f"Original sentence (for embedding): {node.node.text}")
    print(f"Expanded Window Context:\n{node.text}") # This is the 'window'
    print(f"Similarity Score: {node.score:.4f}")

print("\n" + "=" * 50)
print("Final LLM Response:")
print(response)
print("=" * 50)

Run this script: python sentence_window_rag.py

What to Observe: You’ll notice that the Original sentence (for embedding) is very short and precise. However, the Expanded Window Context (which is what the LLM actually receives) is a much larger block of text, containing the original sentence and its surrounding sentences, providing a richer and more coherent context for the LLM to answer the question about deforestation causes. This prevents the LLM from getting a fragmented piece of information.

If you comment out MetadataReplacementPostProcessor(target_metadata_key="window") and rerun, you’ll see the LLM only receives the short, original sentences, leading to potentially less comprehensive answers.

Mini-Challenge: Implement Parent Document Retrieval (Simplified)

You’ve seen Sentence Window Retrieval in action. Now, let’s tackle a simplified version of Parent Document Retrieval.

Challenge: Modify the sentence_window_rag.py script to simulate a basic Parent Document Retrieval strategy.

Here’s the idea:

1. Create “Parent” Chunks: Define larger chunks (e.g., paragraphs) as your parent documents.
2. Create “Child” Chunks: From each parent, create smaller “child” chunks (e.g., sentences).
3. Embed Child Chunks: Embed and store only the child chunks in your vector store, but make sure each child chunk has a reference to its parent document (e.g., an ID or the full parent text).
4. Retrieve and Expand: When a query retrieves a child chunk, instead of just using the child, retrieve its associated parent document and send that larger parent document to the LLM.

Hint:

• You’ll need two different node_parser instances or a custom way to manage parent-child relationships.
• llama-index has a ParentDocumentRetriever (part of llama_index.retrievers) that automates this, but for this challenge, try to implement the logic manually for better understanding.
• Store the full parent text in the metadata of the child nodes. When a child node is retrieved, you can access its metadata to get the full parent document.

What to Observe/Learn:

• How to manage hierarchical relationships between different levels of chunks.
• The benefit of retrieving a broader, more complete context when multiple smaller, related chunks are relevant.
• The trade-offs in complexity versus the quality of context.

Common Pitfalls & Troubleshooting in Context Assembly

Even with advanced techniques, pitfalls can arise:

1. Over-complex Chunking / Diminishing Returns: While advanced methods are powerful, don’t over-engineer. Too many layers of chunking or overly aggressive post-processing can add latency, computational cost, and complexity without proportional gains in answer quality. Always benchmark!
   • Troubleshooting: Start simple, then add complexity incrementally. Evaluate the impact of each new technique on your specific use case and dataset.
2. Misaligned Chunking Strategy with Data/Query: A strategy that works for legal documents might fail for scientific papers or conversational data. For instance, sentence window retrieval might be great for factual questions but less effective for multi-hop reasoning across an entire chapter.
   • Troubleshooting: Understand your data’s structure (e.g., long paragraphs, short bullet points, tables). Analyze your typical user queries. Does the query require fine-grained detail or broad understanding? Tailor your strategy accordingly.
3. Computational Cost and Latency: Advanced techniques like LLM-assisted context structuring or multiple retrieval stages can increase the time it takes to generate a response. Generating summaries or performing reranking adds overhead.
   • Troubleshooting: Monitor latency. Optimize embedding models (e.g., use smaller, faster models if possible). Cache intermediate results. Consider using more powerful hardware or distributed processing for computationally intensive steps. For LLM-assisted structuring, experiment with smaller, faster LLMs for the restructuring task.

Summary: Elevating Your RAG’s Context IQ

Congratulations! You’ve just taken a significant leap in understanding and implementing advanced context assembly for RAG 2.0. We covered:

• The inherent limitations of simple, fixed-size chunking and how it leads to context distortion.
• Sentence Window Retrieval: A technique for precise retrieval combined with rich, surrounding context.
• Auto-Merging/Parent Document Retrieval: For dynamically combining related small chunks into a coherent larger context.
• Hierarchical Chunking: Providing adaptive context at multiple granularities.
• Summary-Based Chunking: Leveraging LLMs to pre-process dense information into concise summaries.
• LLM-Assisted Context Structuring: Using LLMs to actively curate and optimize retrieved information for the final generation.

By mastering these techniques, you’re now equipped to build RAG systems that provide LLMs with truly intelligent, coherent, and highly relevant context, leading to dramatically improved answer quality.

In our next chapter, we’ll dive deeper into Query Rewriting and Transformation, exploring how we can make our queries as intelligent as our context assembly, ensuring we’re asking the right questions in the right way to get the best possible retrieval.

References

• RAG and Generative AI - Azure AI Search - Microsoft Learn
• LlamaIndex Documentation - Sentence Window Retrieval
• LlamaIndex Documentation - Parent Document Retrieval
• LlamaIndex Documentation - Node Parser

This page is AI-assisted and reviewed. It references official documentation and recognized resources where relevant.