Fundamentals · Document Processing & Validation
Document Extraction Workbench
From Documents to Validated Records
I built an invoice extractor that keeps both the value and the evidence for every field. It reads text and scanned documents, validates each record against explicit rules, and sends anything uncertain to a person.
Status: Built
The workbench is built and the code is public. I wrote up the build day by day on the blog, mistakes included.
The Problem
Picture a finance team that copies invoice details into a spreadsheet by hand: vendor, invoice number, date, currency, and total. It is slow, and a single wrong amount is expensive.
An AI model can read the invoice, but a confident wrong answer is worse than a blank one. The team needs a draft record they can check quickly, with the source of every value one click away. The goal is a reviewable draft, not payment approval.
How It Works
Reading a file, recognizing text, extracting meaning, checking a record, and measuring accuracy are separate operations, and the pipeline keeps them separate:
What I Built
- A typed record: Pydantic models for vendor, invoice number, date, currency, and total, each with a quote and page reference
- A reader: Page-by-page text extraction that detects scanned pages instead of silently returning empty text
- A validator: Normalization, quote checks, and review flags, with the reasons recorded
- An evaluator: A labeled dataset and a per-field accuracy report
The first version ran on saved example predictions so the whole pipeline could be tested before any paid model call. Live extraction with a real model came after that.
Design Decisions
Every value carries its evidence
Each field is stored as a value, an exact quote, and a page number, so a reviewer can check the source in seconds.
Missing means null
A field that is absent or ambiguous is null, never guessed. A currency symbol alone is not treated as a currency.
Extraction and normalization are separate
Raw model output is preserved next to the normalized record, so any change can be traced. Money uses Decimal, not floating point.
Business rules live in code
Date, currency, and amount rules are explicit and tested, not left to a sentence in a prompt.
Scans take a separate route
A PDF parser cannot read an image. Scanned pages use a visual route and always require human verification of the evidence.
Documents are untrusted input
The prompt tells the model to ignore instructions written inside an invoice. That is a defense, not a guarantee, so validation still runs afterward.
How I Evaluated It
- •A fresh set of 20 synthetic invoices: 5 clean, 5 with a missing or ambiguous field, 5 that separate subtotal, tax, discount, and final total, and 5 scans or multi-page documents
- •Ten documents for development and ten held out before any tuning
- •Labels written and checked by hand, never produced by the same model that makes the predictions
- •Exact-match accuracy by field, incorrect non-null values, correctly missing fields, review rate, and latency, with mock and live results reported separately
- •A deliberate failure case: a total read as 9.00 instead of 90.00 passes format checks and a quote check, and only the comparison with labeled truth catches it
A low review rate is not the goal on its own: a system can avoid mistakes by sending every document to a person. I report the tradeoff between errors caught and review effort.
Scope
The workbench produces draft records for human inspection. It does not approve or pay invoices, and a record without review flags means only that these particular checks found no issue, not that the invoice is proven correct. It uses synthetic invoices and supports USD, EUR, and GBP, with credit notes out of scope.
