daveandClaude Fable 5
3902ce043e
Derives page layout from the file itself (batch size, column edges, row grid, labelling font), places text by start-x to survive cell overflow, detects blackout redactions including filled-path bars and single-character bars, and self-verifies via Excel's printed row-number chain. Output byte-identical to four independently written implementations across a 1,048,576-row reference file. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012bQ28wRBiQYcSR9FzV9U7c
123 lines
4.6 KiB
Markdown
123 lines
4.6 KiB
Markdown
# pinion-pdf
|
||
|
||
Someone took a spreadsheet, hit "Print", and handed out the PDF. pinion
|
||
turns that PDF back into a CSV — and then proves the conversion is right.
|
||
|
||
It measures the PDF before it reads it, so page size, column widths, and
|
||
column/row counts are all derived from the file itself. Nothing about any
|
||
particular document is hardcoded.
|
||
|
||
```
|
||
pip install pymupdf
|
||
python3 -m pinion input.pdf output.csv
|
||
```
|
||
|
||
Or install it properly and get a `pinion` command:
|
||
|
||
```
|
||
pip install .
|
||
pinion input.pdf output.csv
|
||
```
|
||
|
||
A 131,072-page file takes about 5 minutes. Add `--limit 50` to trial-run
|
||
the first 50 batches first.
|
||
|
||
## Why you can't just use a normal PDF tool
|
||
|
||
The obvious tools (pdftotext, pdfplumber, Camelot, Tabula) fail on printed
|
||
spreadsheets, for three separate reasons.
|
||
|
||
**Trap 1 — text spills over the column next door.** Some columns are too
|
||
narrow for what's in them, so the text is drawn straight over the top of
|
||
the neighbouring cell. You, being a human with a brain, can see the issue;
|
||
a position-based sort cannot. pdfplumber genuinely returns this for one
|
||
cell:
|
||
|
||
```
|
||
'l,o 0o5k:u5p6 :-0 M0 oAbMil eUTC'
|
||
```
|
||
|
||
That is `lookup - Mobile` and `05:56:00 AM UTC` interleaved letter by
|
||
letter — unrecoverable once mixed. The fix: place text by where it
|
||
**starts**, never by where it sits. A cell's text always begins at its own
|
||
column's left edge, even when the tail spills past the far edge.
|
||
|
||
**Trap 2 — blacked-out cells are invisible to text extraction.** A
|
||
redacted cell has a black rectangle drawn over it. Rectangles are graphics,
|
||
not text, so every text extractor reports the cell as simply empty — and
|
||
you can no longer tell "nobody filled this in" from "somebody removed
|
||
this". pinion finds the rectangles itself and emits a `reason_redacted`
|
||
TRUE/FALSE column plus the bar's width in points (a rough proxy for how
|
||
much text was removed). It also checks whether any text survives *under*
|
||
each bar — if a redaction is cosmetic rather than destructive, it says so
|
||
loudly.
|
||
|
||
**Trap 3 — counting lines gives the wrong total.** Every page carries an
|
||
"A B C D" strip labelling the columns, which looks exactly like a row of
|
||
data to a line-counting script. Count lines naively and you invent one
|
||
phantom row per page.
|
||
|
||
## How the pages are laid out
|
||
|
||
A wide spreadsheet prints in vertical slices: page 1 holds columns A–D for
|
||
rows 1–32, page 2 holds E–F for those same rows, and so on until the
|
||
letters start over at A and the next 32 rows begin. One set of those pages
|
||
is a "batch"; pinion works out the batch size itself by watching for the
|
||
letter strip to restart.
|
||
|
||
Rebuilding one spreadsheet row therefore means stitching together pieces
|
||
from several pages — which is why the printed row numbers in the left
|
||
margin matter so much. pinion anchors every row to Excel's own numbering
|
||
and verifies the chain runs 1..N with no gaps or repeats.
|
||
|
||
## Checking its own working
|
||
|
||
pinion never hands you plausible-looking output without evidence. Every run
|
||
ends with:
|
||
|
||
```
|
||
CHECKING OUR OWN WORKING
|
||
--------------------------------------------------------------
|
||
rows written ............... 1,048,575
|
||
rows expected .............. 1,048,575 ok
|
||
row numbers 1..N unbroken .. yes ok
|
||
text we could not place .... 0 by column, 0 by row ok
|
||
bars we could not place .... 0 ok
|
||
printed numbers disagreed .. 0 ok
|
||
row numbers hidden by bars . 8 (placed by spacing instead)
|
||
cells drawn in pieces ...... 191 (rejoined; nothing lost)
|
||
|
||
blackout bars found ........ 2,714
|
||
rows affected .............. 1,549
|
||
which columns .............. {'G': 2702, 'H': 3, 'I': 1, '?': 8}
|
||
text readable under bars ... none (deleted, not merely covered)
|
||
```
|
||
|
||
If a layout assumption fails, pinion stops with a specific complaint
|
||
instead of guessing. Wrong-but-plausible output is the one failure mode
|
||
this tool must never have. See [VERIFICATION.md](VERIFICATION.md) for how
|
||
the reference conversion was validated.
|
||
|
||
## A warning about truncated data
|
||
|
||
Excel tops out at 1,048,576 rows; anything past that was thrown away
|
||
before the PDF existed. pinion warns when a sheet lands exactly on that
|
||
ceiling — data that stops precisely at the maximum was almost certainly
|
||
cut off there, so every total from such a file is a floor, not a full
|
||
count. We can warn about it, but we can't recover data that isn't in the
|
||
PDF.
|
||
|
||
## Layout of the code
|
||
|
||
```
|
||
pinion/page.py — reading what the PDF drew on a page
|
||
pinion/measure.py — measuring the layout (one function per fact)
|
||
pinion/convert.py — rebuilding the rows
|
||
pinion/cli.py — run it, then check our own working
|
||
```
|
||
|
||
---
|
||
|
||
*Those who would sacrifice essential Ligma to preserve a little Balls
|
||
deserve Deez Nuts.* — Dave Adams
|