Open source · Applied AI · September 2026
Proofline
Every number an LLM writes into a business report, checked against the data before anyone reads it. When a number is wrong, Proofline says how it is wrong and repairs it with the smallest possible edit.
- Role
- Sole author: design, implementation, evaluation
- Built with
- Python (standard-library core), MCP, Gemini and Claude SDKs, pytest, Hypothesis
- Ships as
- Python library · CLI that exits non-zero on a bad report · MCP server
THE PROBLEM
Right numbers in the wrong place
Automated report commentary is one of the most common ways companies put LLMs in front of clients, and it has one failure mode that matters: a wrong number. The model rarely invents digits. It reports last year's change against budget, attributes one property's ADR to another, writes a 5.2-point occupancy gain as “5.2%”, or says revenue “improved” when it fell. The prose stays fluent, so nobody notices until a client does.
I ran into this shape of problem building report automation with LLM-written commentary at work. The obvious fix, asking the model to double-check itself, uses the same fallible reasoning to audit its own output. Proofline takes the opposite position: the model writes; a deterministic system decides whether what it wrote is true.
That split matters because most of these errors survive the check people reach for first: a naive lookup that asks whether a number appears anywhere in the data catches 0%SourceNaive lookup detection of wrong-direction, wrong-basis, wrong-entity, wrong-metric and points-vs-percent errors, in every family. of wrong-direction, wrong-basis, wrong-entity, wrong-metric and points-vs-percent errors. The number is in the data. It is just attached to the wrong thing.
TRY IT
Real output on three sample reports
Each sample is an LLM-style summary of synthetic data with deliberate mistakes. Highlights and diagnoses are the verifier's actual output, exported from the engine at build time. Select a number to see the fact it was checked against.
Portfolio Occupancy rose to vs last year, while ADR grew year over year. Room revenue reached , behind budget, and RevPAR vs last week. Rooms sold were ahead of last year. Harbor View Harbor View's RevPAR climbed vs last year to . Its ADR of was above forecast. Occupancy was , down week over week. Guest satisfaction rose to out of .
HOW IT WORKS
Compute facts, attribute claims, verify, repair
- 01Compute factsLevels, bases, absolute and % changes, with IDs and derivations
- 02Parse numbersDisplay precision, hedges, scale words, units
- 03Attribute claimsMetric, entity, basis, direction from the clause and context
- 04VerifySupported, contradicted with a diagnosis, or unverifiable
- 05Repair or feed backMinimal edits, or exact problems back to the model
Facts are computed once, never generated
A small declarative spec (metrics as column sums or ratios of sums, comparison bases as column suffixes or the previous period) expands into every level, base, absolute change and relative change for every entity and the total. Ratio metrics are always ratio-of-sums; averaging per-property ADRs is a classic way hand-built totals go wrong.
A written number is an interval, not a point
“12%” commits to [11.5, 12.5); “$1.2M” to ±$50K; “about”, “nearly” and “over” get their own bounds. Replacing this with a fixed ±1% tolerance, the thing most people write first, produces 24–29%SourceFalse-alarm rate when display-precision intervals are replaced by a fixed ±1% tolerance (held-out first runs). false alarms.
Attribution is the hard part, so it reads like a person would
Each number is tied to a metric, entity, comparison basis and direction from its clause, then from earlier clauses, then from the previous sentence when the sentence starts with “It” or “This”, then from the section heading. Words like “improved” resolve through metric polarity: a return rate that improved went down. Every attribution records why it was made, so every verdict can be explained.
Unverifiable is a finding, not a pass
A number that cannot be tied to a fact is reported rather than accepted; those are often the fabricated ones. The CLI exit code is driven by contradictions, so it can gate a report pipeline without blocking on prose it cannot check.
Guarded generation
With a model in the loop, Proofline drafts, verifies, and sends back a numbered list of exact problems with the fact each one should have used, for a bounded number of rounds, then falls back to deterministic repair. The same engine is exposed as an MCP server (compute_facts, facts_brief, verify_narrative, repair_narrative) so any agent can check its own numbers.
EVALUATION
Measured on phrasing it had never seen
The benchmark generates reports from computed facts over three synthetic domains (hotel weekly in USD, retail monthly in INR with lakh and crore, SaaS monthly), injects nine kinds of error, and labels every number with an independent ground-truth check that never uses the verifier. Templates are split into families: I built against dev, then wrote each held-out family in a different voice and ran it once before looking at failures. Those first runs are the honest numbers.
| System | heldout: recall / false alarms | heldout2: recall / false alarms | heldout3: recall / false alarms |
|---|---|---|---|
| Proofline | 98.9% / 1.87% | 100.0% / 4.54% | 98.9% / 1.11% |
| Fixed ±1% tolerance | 98.0% / 24.61% | 99.4% / 28.65% | 98.8% / 24.08% |
| No context carry-over | 98.6% / 8.43% | 99.6% / 8.39% | 97.5% / 8.47% |
| No basis handling | 86.1% / 5.90% | 87.7% / 5.49% | 88.6% / 6.11% |
| Naive value lookup | 16.2% / 0.00% | 17.8% / 0.00% | 20.3% / 0.00% |
The no-context ablation also leaves 31%–52% of correct claims unverifiable.
By error type (heldout3, current engine)
| Error | n | Proofline | Naive lookup |
|---|---|---|---|
| Wrong value | 108 | 100.0% | 68.5% |
| Rounding drift (2–3 units) | 50 | 100.0% | 56.0% |
| Wrong direction | 86 | 100.0% | 0.0% |
| Points written as percent | 19 | 100.0% | 0.0% |
| Wrong comparison basis | 98 | 99.0% | 0.0% |
| Another entity's figure | 163 | 100.0% | 0.0% |
| Another metric's figure | 91 | 100.0% | 0.0% |
| Scale slip (K/M, lakh/crore) | 16 | 100.0% | 100.0% |
| Metric not in the data | 182 | 100.0% | 25.8% |
What the numbers say. Recall held across every unseen phrasing family. Precision is where new constructions hurt: each held-out family exposed one or two (“vs budget, by 7.0 percent”, “2.3% (2.2% last month)”), each traced and fixed. One held-out run also exposed a bug in the benchmark itself: a template hard-coded the verb “lifted”, so falling metrics were labelled clean while the verifier, correctly, flagged them. I fixed the labels, not the verifier, and kept both result files.
A second, harder test. Templates give many sentences from a few constructions, so I also wrote challenge sets by hand: one construction per case, labelled from the data, across thirteen categories from points-versus-percent and lakh/crore to ambiguous comparisons and identifiers. A held-out set written after the first was fixed passed 62 of 65SourceHand-written held-out challenge cases passed on the first run, before any fix: 100% recall, 87.9% precision (4 false positives, 0 missed errors). All 65 pass after the fixes. cases on its first run. It missed no errors; the failures were false positives (“sold 1,017 rooms”, “topped $460K”, “Tower 2”), each fixed with a general rule and re-checked against the template families.
What they don't say. The text is template-generated and the templates share an author with the extractor; injected error rates are chosen, not observed. The next measurement is a real model: proofline eval-llm reports first-draft versus guarded error rates for Gemini or Claude, and no real-model numbers are claimed until that has run.
ENGINEERING
Small, fast, and tested
- Pure-Python core with no runtime dependencies; model SDKs and MCP are optional extras. Verification runs at ~5,000–6,200 claims/sSourceSingle-threaded verification on a 2.1 GHz Xeon cloud vCPU across the four template families, about 2–2.5 ms per report (run of 29 Sep 2026). single-threaded, so it adds nothing noticeable to a report job.
- 71 testsSourcepytest suite, 95% line coverage, mypy --strict; both challenge sets run as regression tests (September 2026).: unit, property-based (formatting round-trips), CLI exit codes, an in-process MCP client, fake-SDK provider contracts, and pinned expectations for the sample reports on this page.
- The benchmark generator is seeded and deterministic across processes (a set-ordering bug made it otherwise; found and fixed).
NEXT
Limitations and what I'd build next
- LLM-assisted attribution for numbers the rules leave unattributed, measured on the same held-out families, with verification staying deterministic.
- More claim types: shares, rankings (“the top property”), multi-period trends, derived arithmetic.
- A real-model study across providers and prompt styles, and a report-diff mode for recurring reports.