Back to Work

Fundamentals · Document Processing & Validation

Document Extraction Workbench
From Documents to Validated Records

I built an invoice extractor that keeps both the value and the evidence for every field. It reads text and scanned documents, validates each record against explicit rules, and sends anything uncertain to a person.

STRUCTURED EXTRACTIONPYDANTICEVIDENCE TRACKINGVALIDATIONEVALUATION
Document Extraction Workbench

Status: Built

The workbench is built and the code is public. I wrote up the build day by day on the blog, mistakes included.

The Problem

Picture a finance team that copies invoice details into a spreadsheet by hand: vendor, invoice number, date, currency, and total. It is slow, and a single wrong amount is expensive.

An AI model can read the invoice, but a confident wrong answer is worse than a blank one. The team needs a draft record they can check quickly, with the source of every value one click away. The goal is a reviewable draft, not payment approval.

How It Works

Reading a file, recognizing text, extracting meaning, checking a record, and measuring accuracy are separate operations, and the pipeline keeps them separate:

01An invoice arrives as a text file, a text-based PDF, or a scanned PDF
02Python reads each page and flags pages that have no text layer
03Scanned pages route to a vision-capable model instead of text extraction
04The model returns five fields, each with a value, an exact source quote, and a page number
05Normalization code converts dates, currencies, and totals to a fixed format
06Validation checks that each quote appears on its stated page
07Any missing, invalid, or unverifiable field routes the draft record to human review

What I Built

  • A typed record: Pydantic models for vendor, invoice number, date, currency, and total, each with a quote and page reference
  • A reader: Page-by-page text extraction that detects scanned pages instead of silently returning empty text
  • A validator: Normalization, quote checks, and review flags, with the reasons recorded
  • An evaluator: A labeled dataset and a per-field accuracy report

The first version ran on saved example predictions so the whole pipeline could be tested before any paid model call. Live extraction with a real model came after that.

Design Decisions

Every value carries its evidence

Each field is stored as a value, an exact quote, and a page number, so a reviewer can check the source in seconds.

Missing means null

A field that is absent or ambiguous is null, never guessed. A currency symbol alone is not treated as a currency.

Extraction and normalization are separate

Raw model output is preserved next to the normalized record, so any change can be traced. Money uses Decimal, not floating point.

Business rules live in code

Date, currency, and amount rules are explicit and tested, not left to a sentence in a prompt.

Scans take a separate route

A PDF parser cannot read an image. Scanned pages use a visual route and always require human verification of the evidence.

Documents are untrusted input

The prompt tells the model to ignore instructions written inside an invoice. That is a defense, not a guarantee, so validation still runs afterward.

How I Evaluated It

  • •A fresh set of 20 synthetic invoices: 5 clean, 5 with a missing or ambiguous field, 5 that separate subtotal, tax, discount, and final total, and 5 scans or multi-page documents
  • •Ten documents for development and ten held out before any tuning
  • •Labels written and checked by hand, never produced by the same model that makes the predictions
  • •Exact-match accuracy by field, incorrect non-null values, correctly missing fields, review rate, and latency, with mock and live results reported separately
  • •A deliberate failure case: a total read as 9.00 instead of 90.00 passes format checks and a quote check, and only the comparison with labeled truth catches it

A low review rate is not the goal on its own: a system can avoid mistakes by sending every document to a person. I report the tradeoff between errors caught and review effort.

Scope

The workbench produces draft records for human inspection. It does not approve or pay invoices, and a record without review flags means only that these particular checks found no issue, not that the invoice is proven correct. It uses synthetic invoices and supports USD, EUR, and GBP, with credit notes out of scope.

Tech Stack

Python 3.12PydanticpypdfOpenAI APIStructured outputsPDF inputsDecimalPytest