The Hook

You chunk your docs at 512 tokens with 100 overlap. Retrieval precision: 62%. You switch to semantic chunking. Precision: 89%. Fixed-size chunks break semantic boundaries. Semantic chunking respects document structure. The difference is the difference between “RAG works” and “RAG hallucinates.”

The Thesis

Chunking strategy is the single highest-leverage RAG optimization. Fixed-size chunks (512 tokens, 100 overlap) cut sentences, tables, code blocks mid-thought. Semantic chunking (semantic splitter, proposition splitting, LLM-based) preserves semantic units. The precision delta: 25-40% improvement.

The Science

Why Fixed-Size Fails

Problem	Example
Mid-sentence cuts	“The algorithm works by…” [CUT] “…iterating until convergence.”
Table fragmentation	Header in chunk 1, rows in chunk 2
Code fragmentation	Function signature in chunk 1, body in chunk 2
Context loss	“As mentioned above” → context in previous chunk

Research: Zhang et al. (2024, Semantic Chunking for RAG) — 12 datasets, 4 embedding models:

• Fixed 512: 62.3% recall@5
• Semantic: 87.1% recall@5 (+40%)
• Proposition: 89.3% recall@5 (+43%)

Semantic Chunking Methods

Method	Mechanism	Precision Gain	Complexity
Semantic Splitter	Embedding similarity between sentences	+25%	Low
Proposition Splitting	LLM extracts atomic facts	+35%	Medium
LLM-based	LLM segments by topic	+40%	High
Markdown/Structure	Header-aware splitting	+30%	Low

Semantic Splitter (LangChain/llama-index)

Algorithm:

1. Split into sentences
2. Embed each sentence
3. Cluster adjacent sentences by cosine similarity
4. Merge until similarity < threshold

from langchain.text_splitter import SemanticChunker
from sentence_transformers import SentenceTransformer

embedder = SentenceTransformer("BAAI/bge-m3")
chunker = SemanticChunker(
    embedder,
    breakpoint_threshold_type="percentile",
    breakpoint_threshold_amount=95  # top 5% dissimilarity = boundary
)

chunks = chunker.split_text(document)

Proposition Splitting (Microsoft, 2024)

Idea: LLM extracts atomic propositions from text. Each proposition = one chunk.

PROMPT = """
Extract atomic propositions from this text. Each proposition should be:
- Self-contained (understandable alone)
- Single fact/claim
- Include context needed to understand

Text: {text}

Output: JSON list of propositions with context.
"""

Result: Each chunk = one verifiable fact. Retrieval precision ↑, hallucination ↓.

LLM-Based Chunking

PROMPT = """
Segment this document into semantically coherent chunks.
Each chunk should:
- Cover one topic/subtopic
- Be 200-800 tokens
- Preserve complete thoughts
- Include necessary context

Document: {text}

Output: JSON with chunk_text, topic, token_count.
"""

The Comparison

Method	Recall@5	Latency	Cost/1M tokens	Best For
Fixed 512/100	62.3%	0ms	$0	Baseline
Markdown splitter	78.2%	10ms	$0	Structured docs
Semantic Splitter	87.1%	50ms	$0.50	General
Proposition (GPT-3.5)	89.3%	200ms	$2.00	High-stakes
LLM chunking (GPT-4)	91.2%	500ms	$10.00	Critical apps

The Protocol: Choose Your Chunker

Decision Tree

Is document structured (Markdown, HTML, code)?
  YES → Markdown/Header splitter (fast, 80%+ recall)
  NO → Is precision critical? (legal, medical, legal)
    YES → Proposition splitting (GPT-3.5) or LLM chunking
    NO → Semantic Splitter (BGE-M3, 95th percentile)

Implementation Priority

1. Week 1: Replace fixed-size with Markdown/Semantic splitter (5 min change, +25% recall)
2. Week 2: Add proposition splitting for critical collections (legal, medical, technical)
3. Week 3: A/B test chunking strategies on your eval set
4. Week 4: Implement hybrid: semantic for docs, proposition for critical sections

The Counterpoint

• Cost: Proposition/LLM chunking adds $2-10/M tokens
• Latency: LLM chunking adds 200-500ms/doc
• Over-chunking: Too many tiny chunks = retrieval noise
• Evaluation needed: Chunking quality must be measured on YOUR eval set

Practical Takeaway

This week: Replace RecursiveCharacterTextSplitter(512, 100) with SemanticChunker(BGE-M3, 95th percentile). Run your eval set. Expect 25%+ recall improvement in 5 minutes of code change.

---

Sources: Zhang et al. (2024) Semantic Chunking for RAG; LangChain SemanticChunker docs; LlamaIndex SemanticSplitterNodeParser; Microsoft “Proposition-based Retrieval” (2024); RAG evaluation benchmarks (RAGAS, TruLens, 2024).