Building the backend for an AI app in 2026 is different from building a standard CRUD service. You are dealing with streaming responses, vector search, context management, and the latency profile of LLM API calls on top of the normal concerns. The good news is that the right stack is well-understood by now, and you do not need to invent anything.
What changed in 2026
- pgvector became production-grade. PostgreSQL with pgvector handles millions of embeddings reliably. Most AI apps do not need a dedicated vector database.
- Edge runtimes handle streaming. Cloudflare Workers and Vercel Edge Functions both support streaming responses, making low-latency LLM UX possible at the edge.
- LLM APIs standardized on OpenAI-compatible interfaces. Anthropic, Mistral, and most providers expose compatible APIs. Switching models requires one line change in most setups.
- Async by default became universal. FastAPI, Hono, and Bun all handle async natively. Synchronous frameworks are a poor fit for AI backends.
The main stack options
| Stack |
Language |
Best for |
Deploy |
| FastAPI + pgvector |
Python |
ML-heavy, data science adjacent |
Railway, Fly |
| Hono + Drizzle + Postgres |
TypeScript |
Full-stack JS/TS apps |
Vercel, Cloudflare |
| Next.js API routes + Postgres |
TypeScript |
Tight frontend-backend coupling |
Vercel |
| Django + pgvector |
Python |
Teams with existing Django |
Railway, Fly |
| Go + pgx |
Go |
High-throughput inference servers |
Fly, Render |
Streaming: the non-negotiable requirement
Every AI app that calls an LLM must stream responses to the frontend. A 5-second wait for a full response is unusable; streaming the tokens as they arrive is standard UX.
# FastAPI streaming example
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
import anthropic
app = FastAPI()
client = anthropic.Anthropic()
@app.post("/chat")
async def chat(prompt: str):
async def generate():
with client.messages.stream(
model="claude-sonnet-4-5",
max_tokens=1024,
messages=[{"role": "user", "content": prompt}]
) as stream:
for text in stream.text_stream:
yield f"data: {text}\n\n"
return StreamingResponse(generate(), media_type="text/event-stream")
// Hono streaming example
import { Hono } from 'hono'
import { streamSSE } from 'hono/streaming'
import Anthropic from '@anthropic-ai/sdk'
const app = new Hono()
const client = new Anthropic()
app.post('/chat', async (c) => {
const { prompt } = await c.req.json()
return streamSSE(c, async (stream) => {
const response = await client.messages.stream({
model: 'claude-sonnet-4-5',
max_tokens: 1024,
messages: [{ role: 'user', content: prompt }],
})
for await (const chunk of response) {
if (chunk.type === 'content_block_delta') {
await stream.writeSSE({ data: chunk.delta.text ?? '' })
}
}
})
})
Vector storage: when to use what
| Scale |
Recommendation |
| < 1M vectors |
pgvector on Postgres — no extra service |
| 1M–100M vectors |
pgvector with HNSW index or Qdrant |
| > 100M vectors |
Pinecone, Weaviate, or Qdrant managed |
| Need full-text + vector hybrid |
pgvector + pg_trgm or Typesense |
How to pick
- Python team building an AI-first app? FastAPI + Postgres + pgvector. Add Celery for async jobs.
- TypeScript team building a full-stack AI app? Next.js API routes or Hono + Drizzle + Railway Postgres.
- Need edge inference latency? Cloudflare Workers + D1 (SQLite) + Vectorize for vectors.
- Millions of vectors already in production? Evaluate Qdrant managed or Pinecone before migrating from pgvector.
- High-throughput non-streaming inference? Go backend with pgx — handles 10× the concurrency of Python for the same infra cost.
Common mistakes
Blocking on LLM API calls. Synchronous LLM calls block threads for seconds. Use async/await throughout — a blocking call in a FastAPI or Hono handler wastes concurrency.
Not implementing retries. LLM APIs return 429s and 529s under load. Implement exponential backoff with jitter from day one.
Storing embeddings in the application layer. Embeddings belong in a vector-indexed column or table — not in JSON blobs or application memory.
Ignoring context window management. Long conversations accumulate tokens fast. Implement conversation summarization or a sliding window before you hit context limits in production.
What to skip
- Building a custom vector database — pgvector handles millions of vectors with one extension. Custom solutions are maintenance overhead.
- Synchronous Python for AI backends —
requests + Flask blocks under any real concurrency. FastAPI with async is the minimum.
- Storing LLM outputs without structured parsing — use structured outputs (JSON mode or Pydantic) to make LLM responses queryable from day one.
FAQ
Do I need a separate vector database from day one?
No. Start with pgvector. Add a dedicated vector DB when you have specific evidence that pgvector is your bottleneck — most apps never get there.
What is the best ORM for AI apps?
Drizzle (TypeScript) and SQLAlchemy (Python) are both excellent. Drizzle is lighter and faster; SQLAlchemy is more mature for complex queries.
How do I handle LLM rate limits in production?
Use a queue (BullMQ for Node, Celery for Python) for non-interactive requests. For interactive requests, implement client-side retry with exponential backoff.
Is Postgres good enough for high-scale AI apps?
Yes, with proper indexing and connection pooling (pgBouncer or Supavisor). Most AI apps are read-heavy with moderate vector search — Postgres handles this well.
Where to go next
Vector databases in 2026 covers the full vector storage landscape in depth. How to build a RAG app in 2026 shows a complete backend implementation for a retrieval-augmented generation system. Build an app with AI in 2026 covers the full-stack workflow from spec to deployment.