Retrieval-augmented generation (RAG) is the most commonly shipped AI pattern in 2026. It is how you give an LLM access to your data without fine-tuning, and how you ground answers in sources you can verify. The pipeline is straightforward, but the quality lives in the details. This guide covers every step with working Python code and honest guidance on where to invest your effort.
What changed in 2026
- Hybrid search is now the standard — vector-only RAG produces worse results than hybrid dense+sparse search. All production stacks use both.
- Reranking became table stakes — a cross-encoder reranker on the top-20 retrieved chunks before generation significantly improves precision. Worth the ~100ms latency.
- Chunking research matured — the community ran large-scale benchmarks. 500–800 token overlapping chunks with 10–15% overlap are the current best-practice starting point.
- Structured metadata filtering is standard — storing source, date, section, and tenant ID as vector metadata and filtering at query time produces far better precision.
The RAG pipeline
Documents → Chunk → Embed → Store (vector DB)
↓
Query → Embed query → Retrieve top-k → Rerank → Generate
Each step has quality decisions that compound.
Step 1: Chunking
from langchain.text_splitter import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(
chunk_size=700, # tokens
chunk_overlap=70, # ~10% overlap preserves context at boundaries
length_function=len,
separators=["\n\n", "\n", ". ", " ", ""]
)
chunks = splitter.split_text(document_text)
Rules of thumb:
- 500–800 tokens per chunk for most use cases
- 10–15% overlap to avoid cutting sentences at boundaries
- Chunk on semantic boundaries (paragraph > sentence > word)
- Store the source document ID and section with each chunk as metadata
Step 2: Embedding
from anthropic import Anthropic # or openai, cohere, etc.
import openai
client = openai.OpenAI()
def embed_chunks(chunks: list[str]) -> list[list[float]]:
response = client.embeddings.create(
model="text-embedding-3-small", # 1536 dims, good price/quality
input=chunks,
)
return [item.embedding for item in response.data]
Use the same model for indexing and querying. Mixing models produces garbage similarity scores.
Step 3: Storing in pgvector
import psycopg2
import json
conn = psycopg2.connect(DATABASE_URL)
cur = conn.cursor()
cur.execute("""
CREATE TABLE IF NOT EXISTS chunks (
id BIGSERIAL PRIMARY KEY,
doc_id TEXT NOT NULL,
content TEXT NOT NULL,
metadata JSONB,
embedding vector(1536)
)
""")
cur.execute("CREATE INDEX IF NOT EXISTS chunks_embedding_idx ON chunks USING hnsw (embedding vector_cosine_ops)")
for chunk, embedding, meta in zip(chunks, embeddings, metadatas):
cur.execute(
"INSERT INTO chunks (doc_id, content, metadata, embedding) VALUES (%s, %s, %s, %s)",
(meta["doc_id"], chunk, json.dumps(meta), embedding)
)
conn.commit()
Step 4: Retrieval with hybrid search
# Dense vector retrieval
def dense_retrieve(query_embedding, top_k=20):
cur.execute("""
SELECT id, content, metadata,
1 - (embedding <=> %s::vector) AS score
FROM chunks
ORDER BY embedding <=> %s::vector
LIMIT %s
""", (query_embedding, query_embedding, top_k))
return cur.fetchall()
# Add BM25 keyword search for hybrid
# Use pg_trgm for simple hybrid, or a dedicated search layer
Step 5: Reranking
from sentence_transformers import CrossEncoder
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
def rerank(query: str, candidates: list[dict], top_k: int = 5) -> list[dict]:
pairs = [(query, c["content"]) for c in candidates]
scores = reranker.predict(pairs)
ranked = sorted(zip(scores, candidates), reverse=True)
return [c for _, c in ranked[:top_k]]
Step 6: Generation
import anthropic
client = anthropic.Anthropic()
def generate(query: str, context_chunks: list[str]) -> str:
context = "\n\n---\n\n".join(context_chunks)
message = client.messages.create(
model="claude-sonnet-4-5",
max_tokens=1024,
system="Answer the question using only the provided context. If the answer is not in the context, say so.",
messages=[{
"role": "user",
"content": f"Context:\n{context}\n\nQuestion: {query}"
}]
)
return message.content[0].text
How to pick the stack
- Under 10M vectors, running Postgres? pgvector — no extra service needed.
- Need managed fully hosted? Supabase (Postgres + pgvector) or Pinecone.
- 100M+ vectors or complex filtering? Qdrant managed.
- Python team? FastAPI + pgvector + Anthropic SDK.
- TypeScript team? Hono/Next.js + Drizzle + pgvector + Vercel AI SDK.
Common mistakes
Not chunking. Embedding whole documents produces poor retrieval — the signal is diluted across too much text.
Skipping the reranker. Vector similarity alone returns many false positives. A reranker cuts precision errors significantly at low latency cost.
Same prompt for all queries. The generation system prompt should vary by use case. Customer support RAG and internal knowledge base RAG need different instructions.
Not logging retrieval results. Log what was retrieved for every query. This is your primary tool for debugging poor answers.
What to skip
- Fine-tuning instead of RAG for knowledge-intensive tasks — RAG is cheaper, updatable, and auditable.
- Very small chunks (< 100 tokens) — lose too much context; similarity scores become noisy.
- Multi-query fusion without measuring — it adds latency and cost; only add it if you have retrieval benchmarks showing it helps.
FAQ
What is the right chunk size?
Start at 700 tokens with 70-token overlap. Measure recall@5 on a test set. Adjust from there.
Do I need to reembed when I update a document?
Yes — update the chunks and re-embed them. Most systems delete old chunks by doc_id and reinsert.
How do I evaluate RAG quality?
Build a test set of (question, expected_answer, source_document) triples. Measure retrieval recall@k and answer faithfulness (does the answer follow from the retrieved context?).
Can RAG replace fine-tuning?
For knowledge and facts, yes. For behavior, tone, and format — use fine-tuning or system prompts. They solve different problems.
Where to go next
Vector databases in 2026 covers the full storage layer options in depth. Best backend for AI apps in 2026 covers the full server-side stack that surrounds the RAG pipeline. Build an app with AI in 2026 shows how to ship an AI app end-to-end.