Extracting data from documents: what still goes wrong
Accuracy on clean samples is not the number that matters. What matters is what happens when a field is missing.
Clean-sample accuracy is a marketing number
Extraction demos use well-scanned, well-formatted documents. Real inboxes contain photographs taken at an angle, scans with a coffee ring, and a supplier whose layout changed last month without telling anyone.
Design for the failure, not the success
The important behaviour is what the system does when it is unsure. Two designs:
- Return a value with no signal. Somebody downstream trusts it.
- Return a value with a confidence and a pointer to the region it came from. A human checks the low-confidence ones.
The second costs more to build and is the only one that survives contact with real documents.
Validate against what you already know
Much of the value comes from cross-checks, not from the model. Does the line-item total match the stated total? Does the tax figure match the rate for that country? Is the supplier one you have on file? A field that fails arithmetic is wrong regardless of how confident the extraction was.
Keep the original reachable
Always store the source document and the coordinates of every extracted value. When a number is disputed months later, being able to point at the exact region on the exact page ends the conversation quickly.
ToolGo Team
Product & Engineering
We build practical SaaS tools and write about what we learn — from product decisions to technical deep-dives.
