pinion-pdf 1.0.0: rebuild a CSV from a spreadsheet printed to PDF

Derives page layout from the file itself (batch size, column edges, row
grid, labelling font), places text by start-x to survive cell overflow,
detects blackout redactions including filled-path bars and
single-character bars, and self-verifies via Excel's printed row-number
chain. Output byte-identical to four independently written
implementations across a 1,048,576-row reference file.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012bQ28wRBiQYcSR9FzV9U7c

README.md

@@ -0,0 +1,122 @@
+# pinion-pdf
+
+Someone took a spreadsheet, hit "Print", and handed out the PDF. pinion
+turns that PDF back into a CSV — and then proves the conversion is right.
+
+It measures the PDF before it reads it, so page size, column widths, and
+column/row counts are all derived from the file itself. Nothing about any
+particular document is hardcoded.
+
+```
+pip install pymupdf
+python3 -m pinion input.pdf output.csv
+```
+
+Or install it properly and get a `pinion` command:
+
+```
+pip install .
+pinion input.pdf output.csv
+```
+
+A 131,072-page file takes about 5 minutes. Add `--limit 50` to trial-run
+the first 50 batches first.
+
+## Why you can't just use a normal PDF tool
+
+The obvious tools (pdftotext, pdfplumber, Camelot, Tabula) fail on printed
+spreadsheets, for three separate reasons.
+
+**Trap 1 — text spills over the column next door.** Some columns are too
+narrow for what's in them, so the text is drawn straight over the top of
+the neighbouring cell. You, being a human with a brain, can see the issue;
+a position-based sort cannot. pdfplumber genuinely returns this for one
+cell:
+
+```
+'l,o 0o5k:u5p6 :-0 M0 oAbMil eUTC'
+```
+
+That is `lookup - Mobile` and `05:56:00 AM UTC` interleaved letter by
+letter — unrecoverable once mixed. The fix: place text by where it
+**starts**, never by where it sits. A cell's text always begins at its own
+column's left edge, even when the tail spills past the far edge.
+
+**Trap 2 — blacked-out cells are invisible to text extraction.** A
+redacted cell has a black rectangle drawn over it. Rectangles are graphics,
+not text, so every text extractor reports the cell as simply empty — and
+you can no longer tell "nobody filled this in" from "somebody removed
+this". pinion finds the rectangles itself and emits a `reason_redacted`
+TRUE/FALSE column plus the bar's width in points (a rough proxy for how
+much text was removed). It also checks whether any text survives *under*
+each bar — if a redaction is cosmetic rather than destructive, it says so
+loudly.
+
+**Trap 3 — counting lines gives the wrong total.** Every page carries an
+"A B C D" strip labelling the columns, which looks exactly like a row of
+data to a line-counting script. Count lines naively and you invent one
+phantom row per page.
+
+## How the pages are laid out
+
+A wide spreadsheet prints in vertical slices: page 1 holds columns A–D for
+rows 1–32, page 2 holds E–F for those same rows, and so on until the
+letters start over at A and the next 32 rows begin. One set of those pages
+is a "batch"; pinion works out the batch size itself by watching for the
+letter strip to restart.
+
+Rebuilding one spreadsheet row therefore means stitching together pieces
+from several pages — which is why the printed row numbers in the left
+margin matter so much. pinion anchors every row to Excel's own numbering
+and verifies the chain runs 1..N with no gaps or repeats.
+
+## Checking its own working
+
+pinion never hands you plausible-looking output without evidence. Every run
+ends with:
+
+```
+CHECKING OUR OWN WORKING
+--------------------------------------------------------------
+  rows written ............... 1,048,575
+  rows expected .............. 1,048,575   ok
+  row numbers 1..N unbroken .. yes   ok
+  text we could not place .... 0 by column, 0 by row   ok
+  bars we could not place .... 0   ok
+  printed numbers disagreed .. 0   ok
+  row numbers hidden by bars . 8 (placed by spacing instead)
+  cells drawn in pieces ...... 191 (rejoined; nothing lost)
+
+  blackout bars found ........ 2,714
+  rows affected .............. 1,549
+  which columns .............. {'G': 2702, 'H': 3, 'I': 1, '?': 8}
+  text readable under bars ... none (deleted, not merely covered)
+```
+
+If a layout assumption fails, pinion stops with a specific complaint
+instead of guessing. Wrong-but-plausible output is the one failure mode
+this tool must never have. See [VERIFICATION.md](VERIFICATION.md) for how
+the reference conversion was validated.
+
+## A warning about truncated data
+
+Excel tops out at 1,048,576 rows; anything past that was thrown away
+before the PDF existed. pinion warns when a sheet lands exactly on that
+ceiling — data that stops precisely at the maximum was almost certainly
+cut off there, so every total from such a file is a floor, not a full
+count. We can warn about it, but we can't recover data that isn't in the
+PDF.
+
+## Layout of the code
+
+```
+pinion/page.py     — reading what the PDF drew on a page
+pinion/measure.py  — measuring the layout (one function per fact)
+pinion/convert.py  — rebuilding the rows
+pinion/cli.py      — run it, then check our own working
+```
+
+---
+
+*Those who would sacrifice essential Ligma to preserve a little Balls
+deserve Deez Nuts.* — Dave Adams