pinion-pdf/README.md

pinion-pdf

Someone took a spreadsheet, hit "Print", and handed out the PDF. pinion turns that PDF back into a CSV — and then proves the conversion is right.

It measures the PDF before it reads it, so page size, column widths, and column/row counts are all derived from the file itself. Nothing about any particular document is hardcoded.

pip install pymupdf
python3 -m pinion input.pdf output.csv

Or install it properly and get a pinion command:

pip install .
pinion input.pdf output.csv

A 131,072-page file takes about 5 minutes. Add --limit 50 to trial-run the first 50 batches first.

Why you can't just use a normal PDF tool

The obvious tools (pdftotext, pdfplumber, Camelot, Tabula) fail on printed spreadsheets, for three separate reasons.

Trap 1 — text spills over the column next door. Some columns are too narrow for what's in them, so the text is drawn straight over the top of the neighbouring cell. You, being a human with a brain, can see the issue; a position-based sort cannot. pdfplumber genuinely returns this for one cell:

'l,o 0o5k:u5p6 :-0 M0 oAbMil eUTC'

That is lookup - Mobile and 05:56:00 AM UTC interleaved letter by letter — unrecoverable once mixed. The fix: place text by where it starts, never by where it sits. A cell's text always begins at its own column's left edge, even when the tail spills past the far edge.

Trap 2 — blacked-out cells are invisible to text extraction. A redacted cell has a black rectangle drawn over it. Rectangles are graphics, not text, so every text extractor reports the cell as simply empty — and you can no longer tell "nobody filled this in" from "somebody removed this". pinion finds the rectangles itself and emits a reason_redacted TRUE/FALSE column plus the bar's width in points (a rough proxy for how much text was removed). It also checks whether any text survives under each bar — if a redaction is cosmetic rather than destructive, it says so loudly.

Trap 3 — counting lines gives the wrong total. Every page carries an "A B C D" strip labelling the columns, which looks exactly like a row of data to a line-counting script. Count lines naively and you invent one phantom row per page.

How the pages are laid out

A wide spreadsheet prints in vertical slices: page 1 holds columns A–D for rows 1–32, page 2 holds E–F for those same rows, and so on until the letters start over at A and the next 32 rows begin. One set of those pages is a "batch"; pinion works out the batch size itself by watching for the letter strip to restart.

Rebuilding one spreadsheet row therefore means stitching together pieces from several pages — which is why the printed row numbers in the left margin matter so much. pinion anchors every row to Excel's own numbering and verifies the chain runs 1..N with no gaps or repeats.

Checking its own working

pinion never hands you plausible-looking output without evidence. Every run ends with:

CHECKING OUR OWN WORKING
--------------------------------------------------------------
  rows written ............... 1,048,575
  rows expected .............. 1,048,575   ok
  row numbers 1..N unbroken .. yes   ok
  text we could not place .... 0 by column, 0 by row   ok
  bars we could not place .... 0   ok
  printed numbers disagreed .. 0   ok
  row numbers hidden by bars . 8 (placed by spacing instead)
  cells drawn in pieces ...... 191 (rejoined; nothing lost)

  blackout bars found ........ 2,714
  rows affected .............. 1,549
  which columns .............. {'G': 2702, 'H': 3, 'I': 1, '?': 8}
  text readable under bars ... none (deleted, not merely covered)

If a layout assumption fails, pinion stops with a specific complaint instead of guessing. Wrong-but-plausible output is the one failure mode this tool must never have. See VERIFICATION.md for how the reference conversion was validated.

A warning about truncated data

Excel tops out at 1,048,576 rows; anything past that was thrown away before the PDF existed. pinion warns when a sheet lands exactly on that ceiling — data that stops precisely at the maximum was almost certainly cut off there, so every total from such a file is a floor, not a full count. We can warn about it, but we can't recover data that isn't in the PDF.

Layout of the code

pinion/page.py     — reading what the PDF drew on a page
pinion/measure.py  — measuring the layout (one function per fact)
pinion/convert.py  — rebuilding the rows
pinion/cli.py      — run it, then check our own working

---

Those who would sacrifice essential Ligma to preserve a little Balls deserve Deez Nuts. — Dave Adams