daveandClaude Fable 5
3902ce043e
Derives page layout from the file itself (batch size, column edges, row grid, labelling font), places text by start-x to survive cell overflow, detects blackout redactions including filled-path bars and single-character bars, and self-verifies via Excel's printed row-number chain. Output byte-identical to four independently written implementations across a 1,048,576-row reference file. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012bQ28wRBiQYcSR9FzV9U7c
3.3 KiB
How the reference conversion was validated
pinion was developed against a real file: a 131,072-page, 360 MB PDF of a law-enforcement network-audit spreadsheet (1,048,576 rows × 14 columns, printed from Excel via PDF-XChange on Windows). Every design decision below was forced by something that actually went wrong on that file.
Independent checks the output passed
-
Row-number chain. Excel prints its own row numbers in the left margin. The reconstruction reproduces 1..1,048,576 with no gaps and no duplicates. Any method that drops or invents rows fails this immediately — a competing hand count of the same file reported 1,081,343 rows, which is exactly one phantom row per page (Trap 3) and also exceeds Excel's hard row limit, so it can be rejected on arithmetic alone.
-
Cross-engine spot check. 40 randomly sampled rows were re-extracted with pdftotext (Poppler — a completely independent PDF engine) and matched on ID, agency, and printed row number: 40/40.
-
Implementation-independent reproduction. Four separately written implementations (a hardcoded-geometry minimal script, a heavily commented teaching version, a parallel self-verifying version, and this package) produce byte-identical CSVs across all 1,048,575 data rows, verified with
cmp. -
Internal consistency. Every value in the ID column is a well-formed UUID (1,048,575 of 1,048,575); every row has exactly the same field count; zero text runs failed to map to a row or column.
Bugs the checks caught during development
These are the reasons the self-checks exist. Each was caught only by diffing against known-good output or by a check line going red:
- Single-character redactions are ~3.4pt wide — narrower than some grid marks. A size-based bar filter dropped 5 of them. Fix: filter by shape (taller than half a row, thicker than a hairline), not by area.
- Some blackout bars are filled four-sided paths, not rectangles. Visually identical, different PDF instructions. Inspecting individual line segments missed 4 bars, including one 121pt wide. Fix: measure whole filled regions, never segments.
- Two redaction bars cover the printed row number itself, making the
gutter unreadable on those pages. Anchoring rows solely on printed
numbers silently dropped both rows. Fix: anchor on geometry, use the
printed numbers as a cross-check (
printed numbers disagreedmust be 0). - A one-character join bug (
" ".joinvs"".join) corrupted the 191 cells whose text the PDF drew in multiple pieces. Caught by byte-diff.
Known artifacts in the source data (not conversion errors)
- 19 rows contain an unmapped
ffligature (the subset font has no Unicode mapping for it):Sex O�enses,Tra�ic. Exact-match filters on those strings will miss these rows. - 141 rows contain double-encoded UTF-8 from the source system
(
100’s,–). Present in the PDF itself. - The sheet ends at exactly 1,048,576 rows — Excel's ceiling — with the final row fully populated. Comparison of per-day record counts against the file's claimed date range shows several end-of-range days thinned or absent: the export was truncated at the ceiling, and roughly 17% of the original records are estimated to be missing. All totals from the file are floors.