# pinion-pdf

Someone took a spreadsheet, hit "Print", and handed out the PDF. pinion
turns that PDF back into a CSV — and then proves the conversion is right.

It measures the PDF before it reads it, so page size, column widths, and
column/row counts are all derived from the file itself. Nothing about any
particular document is hardcoded.

```
pip install pymupdf
python3 -m pinion input.pdf output.csv
```

Or install it properly and get a `pinion` command:

```
pip install .
pinion input.pdf output.csv
```

A 131,072-page file takes about 5 minutes. Add `--limit 50` to trial-run
the first 50 batches first.

## Why you can't just use a normal PDF tool

The obvious tools (pdftotext, pdfplumber, Camelot, Tabula) fail on printed
spreadsheets, for three separate reasons.

**Trap 1 — text spills over the column next door.** Some columns are too
narrow for what's in them, so the text is drawn straight over the top of
the neighbouring cell. You, being a human with a brain, can see the issue;
a position-based sort cannot. pdfplumber genuinely returns this for one
cell:

```
'l,o 0o5k:u5p6 :-0 M0 oAbMil eUTC'
```

That is `lookup - Mobile` and `05:56:00 AM UTC` interleaved letter by
letter — unrecoverable once mixed. The fix: place text by where it
**starts**, never by where it sits. A cell's text always begins at its own
column's left edge, even when the tail spills past the far edge.

**Trap 2 — blacked-out cells are invisible to text extraction.** A
redacted cell has a black rectangle drawn over it. Rectangles are graphics,
not text, so every text extractor reports the cell as simply empty — and
you can no longer tell "nobody filled this in" from "somebody removed
this". pinion finds the rectangles itself and emits a `reason_redacted`
TRUE/FALSE column plus the bar's width in points (a rough proxy for how
much text was removed). It also checks whether any text survives *under*
each bar — if a redaction is cosmetic rather than destructive, it says so
loudly.

**Trap 3 — counting lines gives the wrong total.** Every page carries an
"A B C D" strip labelling the columns, which looks exactly like a row of
data to a line-counting script. Count lines naively and you invent one
phantom row per page.

## How the pages are laid out

A wide spreadsheet prints in vertical slices: page 1 holds columns A–D for
rows 1–32, page 2 holds E–F for those same rows, and so on until the
letters start over at A and the next 32 rows begin. One set of those pages
is a "batch"; pinion works out the batch size itself by watching for the
letter strip to restart.

Rebuilding one spreadsheet row therefore means stitching together pieces
from several pages — which is why the printed row numbers in the left
margin matter so much. pinion anchors every row to Excel's own numbering
and verifies the chain runs 1..N with no gaps or repeats.

## Checking its own working

pinion never hands you plausible-looking output without evidence. Every run
ends with:

```
CHECKING OUR OWN WORKING
--------------------------------------------------------------
  rows written ............... 1,048,575
  rows expected .............. 1,048,575   ok
  row numbers 1..N unbroken .. yes   ok
  text we could not place .... 0 by column, 0 by row   ok
  bars we could not place .... 0   ok
  printed numbers disagreed .. 0   ok
  row numbers hidden by bars . 8 (placed by spacing instead)
  cells drawn in pieces ...... 191 (rejoined; nothing lost)

  blackout bars found ........ 2,714
  rows affected .............. 1,549
  which columns .............. {'G': 2702, 'H': 3, 'I': 1, '?': 8}
  text readable under bars ... none (deleted, not merely covered)
```

If a layout assumption fails, pinion stops with a specific complaint
instead of guessing. Wrong-but-plausible output is the one failure mode
this tool must never have. See [VERIFICATION.md](VERIFICATION.md) for how
the reference conversion was validated.

## A warning about truncated data

Excel tops out at 1,048,576 rows; anything past that was thrown away
before the PDF existed. pinion warns when a sheet lands exactly on that
ceiling — data that stops precisely at the maximum was almost certainly
cut off there, so every total from such a file is a floor, not a full
count. We can warn about it, but we can't recover data that isn't in the
PDF.

## Layout of the code

```
pinion/page.py     — reading what the PDF drew on a page
pinion/measure.py  — measuring the layout (one function per fact)
pinion/convert.py  — rebuilding the rows
pinion/cli.py      — run it, then check our own working
```

---

*Those who would sacrifice essential Ligma to preserve a little Balls
deserve Deez Nuts.* — Dave Adams