A vendor says their extraction is 99 percent accurate. That number is meaningless without three pieces of context: accurate at what granularity, on what documents, and counting what as an error. Change any of those and the same system reports anywhere from 99 to 70 percent.
Getting this right matters because the number determines your staffing. Extraction that needs human review on a tenth of documents is a different business case from one needing review on half.
What changed in 2026
- Vision models displaced dedicated pipelines for many cases. General multimodal models became competitive with purpose-built extraction stacks on common document types, which changed the build-versus-buy calculation.
- Confidence calibration became a differentiator. Systems that reliably know when they are unsure proved more valuable than systems with marginally higher raw accuracy.
- Document-level reporting became the norm. Buyer pressure pushed vendors toward reporting the metric that predicts operational cost rather than the flattering one.
- Straight-through processing rate emerged as the headline. The percentage of documents requiring no human touch became the number operations teams actually track.
Why field-level accuracy misleads
| Fields per document |
Per-field accuracy |
Documents fully correct |
| 5 |
99% |
About 95% |
| 10 |
99% |
About 90% |
| 20 |
99% |
About 82% |
| 20 |
98% |
About 67% |
| 20 |
95% |
About 36% |
Errors compound multiplicatively across fields. A system with genuinely excellent per-field accuracy can still require human review on most documents once the field count rises, and that is the number that determines headcount.
This is not a criticism of vendors quoting field accuracy — it is a real metric. It is a warning that it does not translate to operational cost in the way people assume.
Building an honest measurement
Take a hundred documents from your real pipeline, weighted toward the difficult ones — the scans, the unusual layouts, the vendors with idiosyncratic templates. A test set of clean documents measures the easy case and tells you nothing about your actual error rate.
Label them by hand. This is tedious and it is the only way to have ground truth. Record every field's correct value including the tricky ones: dates in ambiguous formats, amounts with unusual separators, fields that are legitimately absent.
Then measure three things. Field-level accuracy, to compare systems. Document-level accuracy, to predict review load. And confidence calibration — when the system reports high confidence, how often is it right? A well-calibrated system lets you auto-approve the confident majority and route the rest to a human, which is where the actual automation value is.
Watch for the difference between a wrong value and a missing one. A blank field is caught by validation; a confidently wrong value flows downstream into your accounting system. Weight those differently in your evaluation because they cost differently. The pipeline architecture for layout-heavy documents is covered in multimodal RAG explained, and schema enforcement in constrained decoding explained.
Common mistakes
- Testing on clean documents. Your error rate lives in the messy tail.
- Comparing vendors on their own reported numbers. Different metrics, different test sets, not comparable.
- Ignoring confidence calibration. Without it you cannot route to review intelligently, and you lose most of the automation benefit.
- Treating all fields equally. An error in an invoice total costs more than an error in a description field.
- Not re-testing after a model change. Extraction quality shifts with model updates, sometimes in both directions.
FAQ
What is a realistic document-level accuracy?
It depends entirely on document quality and field count. Clean digital forms with few fields reach very high rates; scanned multi-vendor invoices with twenty fields are considerably lower. Measure yours rather than benchmarking against a general figure.
Should I use a general vision model or a specialist tool?
General models are increasingly competitive and more flexible for varied layouts. Specialist tools often bring better calibration, validation rules, and review workflows. Test both on your documents.
How do I handle fields that are legitimately missing?
Distinguish absent from missed in your ground truth and your metrics, or your accuracy number is measuring the wrong thing.
Does more prompting improve extraction?
Providing the schema, field descriptions, and a worked example helps meaningfully. Beyond that, gains come from document quality and model capability rather than prompt length.
Where to go next
For the retrieval side of document AI, read multimodal RAG explained. For guaranteeing output shape, constrained decoding explained, and for the models involved, vision language models explained.