Text-to-SQL went from "amusing demo" to "real production tool" in 2026 — for users who understand the failure modes. The accuracy gap between specialized tools (Vanna, Sherlock) and general LLMs has narrowed; frontier models now reach 75-85% on real-world schemas, specialized tools push 90%+. Here is the honest comparison.
What changed in 2026
- Frontier model accuracy on text-to-SQL benchmarks crossed 80%. Spider 2.0 and BIRD show GPT-5 and Opus 4.7 in the 80-85% range.
- Specialized tools added schema-aware fine-tuning. Vanna and Sherlock train on your schema for 90%+ accuracy.
- Reasoning modes help on complex joins. Multi-table queries with subqueries benefit most from o1/Opus thinking modes.
Real-world accuracy benchmark
We benchmarked five tools on a real e-commerce schema (47 tables, 312 columns, complex naming) with 100 questions ranging from "how many orders today?" to "show me LTV by acquisition cohort with retention curve."
| Tool |
Easy questions |
Medium |
Complex |
Overall |
| GPT-5 (raw) |
95% |
78% |
52% |
75% |
| Opus 4.7 (raw) |
96% |
80% |
58% |
78% |
| GPT-5 + schema RAG |
97% |
85% |
68% |
83% |
| Vanna AI |
96% |
88% |
76% |
87% |
| Defog Sherlock |
97% |
91% |
78% |
89% |
The 14-point gap between raw GPT-5 and Sherlock comes from schema-aware fine-tuning, not model quality.
Vanna AI
Open-source-friendly text-to-SQL with RAG over your schema and example queries. The "training" loop is providing examples; quality improves with more examples.
Cost: Free open-source; SaaS $39/mo Pro, $99/mo Team.
Best at: technical teams willing to train on their schema, lower-budget production text-to-SQL.
Sharp edge: training requires effort; first-day quality is just-okay. Quality climbs with examples.
Defog Sherlock
Specialized text-to-SQL with schema understanding from documentation, foreign keys, and cardinality stats. Highest accuracy in our test on complex queries.
Cost: SaaS $50-200/mo per user.
Best at: mid-to-large analytics teams, complex schemas, where 90%+ accuracy is required.
Sharp edge: more complex setup; pays back at scale, less so for solo work.
SQLChat
Cloud-based chat-with-SQL interface with reasonable defaults. Less customizable than Vanna or Sherlock; faster to start.
Cost: Free tier; $10-30/mo Pro.
Best at: small teams, simple-to-medium schemas, quick analyses.
When to use frontier LLMs directly
For ad-hoc analysis on schemas you understand, GPT-5 or Opus 4.7 with the schema pasted in works fine. Provide:
- Table names and key columns
- Foreign key relationships
- 2-3 example queries
Accuracy at 80%+ is real. The other 20% is where you need to read the query before running.
Common pitfalls
Ambiguous column names. status, type, state columns kill accuracy. AI guesses semantics from name; if status could mean order_status or payment_status, it'll guess wrong.
Date column handling. "Last 30 days" can mean inclusive or exclusive of today; AI tools handle this inconsistently. Specify explicitly.
Aggregation defaults. "Show revenue" — sum or average? Per period or total? Be explicit.
Joins on weak keys. AI tools sometimes join on columns with the same name but different meaning. Foreign-key annotations in your schema metadata help.
Workflow that works
- Always EXPLAIN before running. Read the SQL; verify the joins.
- Test on a sample. Run with
LIMIT 100 first.
- Compare to manually-written. For critical queries, have an analyst verify.
- Build an example library. Save the questions/queries you ask repeatedly; they become RAG context.
When text-to-SQL fails
Multi-step business logic. "Show me customers who churned but came back" requires defining churn, defining return, joining sequence. AI gets the SQL but the business definitions might not match yours.
Performance optimization. AI generates correct queries; not always efficient ones. EXPLAIN ANALYZE before running on prod-sized data.
Cross-database queries. AI tools handle single warehouse well; cross-warehouse federation is rougher.
FAQ
Should I let non-technical users run AI SQL on prod?
With guardrails — read-only role, query timeout, no DELETE/UPDATE. With those, yes.
What about NL-to-SQL inside BI tools (Tableau, Looker)?
Looker's "Explore Assistant" is solid in 2026. Tableau Pulse is competent. Both leverage their data model context, which helps accuracy.
Is fine-tuning worth it?
For schemas with >100 tables and recurring complex queries, yes. For schemas under 30 tables, RAG over schema docs is usually enough.
Where to go next
For related guides see AI data analyst tools in 2026, AI spreadsheet tools in 2026, and Best RAG tools for production in 2026.