Day 3 of Document Extraction Workbench: A page I can read can still have no text
Day 3 was the first real code change. I taught the program to read an invoice file. A normal PDF came back with the vendor and the total. A scanned PDF came back empty, even though I can read that page myself.
The short version
On Day 1 the rule was: if the page never gives a fact, the value stays null. On Day 2 that rule got a shape: a value, a quote, and a page number.
Day 3 is the step before any of that. The program has to get the words off the file. I still did not call a model. I typed nothing from the page this time. read_pages did the reading.
What the reader returns
read_pages lives in work/extract.py. A text file counts as one page. A PDF is one string per page. extract_text() reads words already stored in the PDF. It does not look at a picture. A page with no stored words becomes "".
def read_pages(path):
path = Path(path)
if path.suffix.lower() == ".txt":
return [path.read_text()]
if path.suffix.lower() != ".pdf":
raise ValueError("Use a txt or pdf file")
return [page.extract_text() or "" for page in PdfReader(path).pages]- The Northstar text file came back as a list of one string: vendor, invoice number, date, currency, and
125.00. - The Northstar PDF came back with the same facts, plus a title line,
Training invoice. The spacing was different. The words were stored in the file, so the reader could see them. - The scan came back as:
['']That is one page, and the string is empty. Open invoice-scan.pdf and the invoice is still there. The file is a picture of a page. This function only reads stored text.
What broke on the way there
The first PDF run crashed with NameError: name 'PdfReader' is not defined. The next run crashed with NameError: name 'Path' is not defined.
Save is set up to delete any import the file does not use yet. I added PdfReader and Path before the function used them, saved, and those lines disappeared. The text file still worked, because reading a .txt file only needs Path, and that import was still there for the first run. The PDF line calls PdfReader, so it crashed. Then Path was gone too, and all three commands crashed.
The imports stayed after I put them back in the same save as the function that uses them. OpenAI and Decimal are still absent on purpose. Later lessons need them, and the formatter will keep them once a line of code uses them.
What I set up on Day 3
Day 3 was two commits.
feat: read text and pdf pages from a file.read_pagesnow reads a text file and a text PDF.docs: updated notes file. The journal now uses the same three boxes as my other project: what I did, what broke, and what I learned.
Frequently asked questions
What does read_pages return?
A list of strings, one string per page. A text file is one page in this project.
Why did the scan come back empty?
The page is a picture. extract_text() reads text stored in the PDF. It does not recognize characters in an image. A later lesson handles pictures.
Does a PDF I can open mean the program can read it?
Only when the file contains stored text. I can read the scan. The reader returned an empty string.
Did a model read the invoice today?
No. The program read stored text only. The scan is still waiting for a later lesson.
Next up
The scan is still unread. I will write about what I learn as I go, including the mistakes.
Related: Day 2 of Document Extraction Workbench: A missing fact is null, not blank and not zero
