pinion-pdf/README.md

# pinion-pdf

*Those who would sacrifice essential Balls to preserve a little Ligma deserve Deez Nuts.* — Dave Adams

Someone took a spreadsheet, hit "Print", and handed out the PDF. pinion
turns that PDF back into a CSV — and then proves the conversion is right.

It measures the PDF before it reads it, so page size, column widths, and
column/row counts are all derived from the file itself. Nothing about any
particular document is hardcoded.

```
pip install pymupdf
python3 -m pinion input.pdf output.csv
```

Or install it and get a `pinion` command:

```
pip install .
pinion input.pdf output.csv
```

A 131,072-page file takes about 5 minutes. Add `--limit 50` to trial-run
the first 50 batches first.

## Why you can't just use a normal PDF tool

The obvious tools (pdftotext, pdfplumber, Camelot, Tabula) have some common
trappings.

### Trap 1 — Text Spillover
Some columns can be too narrow for what's in them, and text spills into neighbors.
You, being a human with a brain, can see the issue - OCR or position-based sort cannot.

Something like `pdfplumber` will interleave spillover text, giving you unrecoverable nonsense.

Our approach measure start points and then uses the PDF's embedded information about write
order to trace the text.

### Trap 2 - Redactions Get Lost
Blacked out cells (which are graphics) can be invisible to text extractors.

This might conflate 'redacted' with 'empty', but we generally want to preserve this context.

Our approach searches for redaction boxes, tracks their size, and adds some extra csv fields
in case this information is relevant later. It also picks up on bad redactions
(text survival under the boxes).


### Trap 3 - Line Counts are Janky

Every page in the typical export format carries an "A B C D ..." strip labeling columns.

To a eegular old parser, that's a row...but it's obviously not.

We avoid phantom rows by stripping those tracking rows from the output.


## Two print formats

pinion sniffs page 1 and picks the right strategy on its own:

**Gridded** — Excel printed with row/column headings on. Printed row
numbers and the letter strip anchor everything; this is the mode described
below.

**Headerless** — the same audits printed via "Microsoft Print to PDF" with
headings off. No gridlines, no row numbers, adjacent cells fused into
single text runs, cells wrapping mid-value, and a schema that drifts
between exports (one month has a License Plate column, the next doesn't).
Here pinion anchors rows on the UUID that starts every record and takes
the column map from the file's own page-1 header labels. Columns absent
from a given export are reported loudly and left empty — never guessed.

If a file is neither, pinion says so and stops.

## How the pages are laid out

A wide spreadsheet prints in vertical slices. In the original data used to build this tool,
page 1 held columns A-D for 32 rows, page 2 held E-F, etc. 

For nomenclature, one set of those pages make a 'batch'. The pattern then repeats.

Rebuilding a row naturally requires stiching together several pages.

We track our calculated rows against the reported row numbers to make sure we haven't
lost the plot as we process things. Any misses or phantoms -> explosion.

## How We Verify Accuracy

Every run contains internal consistency checks to make sure things haven't gone off the rails.

We track and log any row number deviation, unplaceable text, unplaceable redaction bars,
and any time we made an assumption regarding a row number, etc because of a redaction bar overlap.

If a layout assumption fails during processing, we stop and yell about it.


```
OUTPUT VALIDATION
--------------------------------------------------------------
  rows written ............... 1,048,575
  rows expected .............. 1,048,575   ok
  row numbers 1..N unbroken .. yes   ok
  text we could not place .... 0 by column, 0 by row   ok
  bars we could not place .... 0   ok
  printed numbers disagreed .. 0   ok
  row numbers hidden by bars . 8 (placed by spacing instead)
  cells drawn in pieces ...... 191 (rejoined; nothing lost)

  blackout bars found ........ 2,714
  rows affected .............. 1,549
  which columns .............. {'G': 2702, 'H': 3, 'I': 1, '?': 8}
  text readable under bars ... none
```

If a layout assumption fails, pinion stops with a specific complaint
instead of guessing. Wrong-but-plausible output is the one failure mode
this tool must never have. See [VERIFICATION.md](VERIFICATION.md) for how
the reference conversion was validated.

## Data Truncation Warning

Excel tops out at 1,048,576 rows.

If you get a result with 1,048,575 rows + a header row, you must either immediately purchase
a lottery ticket or consider the possibility of data truncation.

We issue a warning, but the truncated data is an issue in the source PDF.

## General Code Layout

```
pinion/page.py       — Read the PDF page layout
pinion/measure.py    — Measure the gridded layout
pinion/convert.py    — Rebuilt rows using our measurements
pinion/headerless.py — The no-headings variant (UUID + header-label anchors)
pinion/cli.py        — Format sniffing, execution harness + inline sanity checks
```