daveandClaude Fable 5
74b06be742
Derives page layout from the file itself (batch size, column edges, row grid, heading font), places text by start-x to survive cell overflow, detects blackout redactions including filled-path bars and single-character bars, and self-verifies via Excel's printed row-number chain. Output byte-identical to four independently written implementations across a 1,048,576-row reference file. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012bQ28wRBiQYcSR9FzV9U7c
64 lines
3.3 KiB
Markdown
64 lines
3.3 KiB
Markdown
# How the reference conversion was validated
|
||
|
||
pinion was developed against a real file: a 131,072-page, 360 MB PDF of a
|
||
law-enforcement network-audit spreadsheet (1,048,576 rows × 14 columns,
|
||
printed from Excel via PDF-XChange on Windows). Every design decision below
|
||
was forced by something that actually went wrong on that file.
|
||
|
||
## Independent checks the output passed
|
||
|
||
1. **Row-number chain.** Excel prints its own row numbers in the left
|
||
margin. The reconstruction reproduces 1..1,048,576 with no gaps and no
|
||
duplicates. Any method that drops or invents rows fails this
|
||
immediately — a competing hand count of the same file reported
|
||
1,081,343 rows, which is exactly one phantom row per page (Trap 3) and
|
||
also exceeds Excel's hard row limit, so it can be rejected on
|
||
arithmetic alone.
|
||
|
||
2. **Cross-engine spot check.** 40 randomly sampled rows were re-extracted
|
||
with pdftotext (Poppler — a completely independent PDF engine) and
|
||
matched on ID, agency, and printed row number: 40/40.
|
||
|
||
3. **Implementation-independent reproduction.** Four separately written
|
||
implementations (a hardcoded-geometry minimal script, a heavily
|
||
commented teaching version, a parallel self-verifying version, and this
|
||
package) produce **byte-identical** CSVs across all 1,048,575 data
|
||
rows, verified with `cmp`.
|
||
|
||
4. **Internal consistency.** Every value in the ID column is a
|
||
well-formed UUID (1,048,575 of 1,048,575); every row has exactly the
|
||
same field count; zero text runs failed to map to a row or column.
|
||
|
||
## Bugs the checks caught during development
|
||
|
||
These are the reasons the self-checks exist. Each was caught only by
|
||
diffing against known-good output or by a check line going red:
|
||
|
||
- **Single-character redactions are ~3.4pt wide** — narrower than some
|
||
grid marks. A size-based bar filter dropped 5 of them. Fix: filter by
|
||
shape (taller than half a row, thicker than a hairline), not by area.
|
||
- **Some blackout bars are filled four-sided paths, not rectangles.**
|
||
Visually identical, different PDF instructions. Inspecting individual
|
||
line segments missed 4 bars, including one 121pt wide. Fix: measure
|
||
whole filled regions, never segments.
|
||
- **Two redaction bars cover the printed row number itself**, making the
|
||
gutter unreadable on those pages. Anchoring rows solely on printed
|
||
numbers silently dropped both rows. Fix: anchor on geometry, use the
|
||
printed numbers as a cross-check (`printed numbers disagreed` must be 0).
|
||
- **A one-character join bug** (`" ".join` vs `"".join`) corrupted the 191
|
||
cells whose text the PDF drew in multiple pieces. Caught by byte-diff.
|
||
|
||
## Known artifacts in the source data (not conversion errors)
|
||
|
||
- 19 rows contain an unmapped `ff` ligature (the subset font has no
|
||
Unicode mapping for it): `Sex O�enses`, `Tra�ic`. Exact-match filters on
|
||
those strings will miss these rows.
|
||
- 141 rows contain double-encoded UTF-8 from the source system
|
||
(`100’s`, `–`). Present in the PDF itself.
|
||
- The sheet ends at exactly 1,048,576 rows — Excel's ceiling — with the
|
||
final row fully populated. Comparison of per-day record counts against
|
||
the file's claimed date range shows several end-of-range days thinned or
|
||
absent: the export was truncated at the ceiling, and roughly 17% of the
|
||
original records are estimated to be missing. All totals from the file
|
||
are floors.
|