LLMs are fundamentally probabilistic. Ask one the same question twice and you can get two different answers — that's not a bug, it's how sampling works. Most of the time that's fine. It is not fine when the question is "is this a pre-money or post-money valuation" and the answer decides who owns what in a startup financing. Finance needs determinism; LLMs supply probability. Nobody was measuring that gap, so Probity does: it benchmarks how often a model's answer wobbles on real term sheets, charters, SAFEs, convertible notes, and cap tables — before you ever get to whether the answer is right.
- Wobble (the core metric) — does the model give the same answer when you ask it the same question 20 times? A model whose answer flips run to run cannot be trusted in a workflow that touches money, even when it is often right. This is label-free: it needs no ground truth, only repetition.
- Accuracy — does the model get the answer right, graded against a validated answer that a human extracted from the source document (not authored by an AI)?
These are scored separately and never averaged into one headline — a model can be perfectly consistent and consistently wrong. Models are run across a size ladder (1B local through hosted frontier) to test whether wobble falls as capability rises, via OpenRouter and direct provider APIs.
Every model is measured at two temperatures — 0.7 and 0.1 — over the identical tests, items, prompts and scorer, so the two arms compare as a paired difference with temperature as the only variable. See the paired comparison; the short version is that lowering temperature is not a general fix for wobble (below).
pip install probity-bench
probity-bench onboard # pick documents to fetch, models to run, and store your API key(s)onboard is a guided wizard — same idea as openclaw onboard or claude setup: it walks you
through which leaves to pull real SEC documents for, which models to benchmark (auto-detects local
Ollama models; DeepSeek/Gemini for hosted), and collects + verifies any API key by making one
real call before it lets you proceed. Everything is stored locally at ~/.probity/ — nothing
leaves your machine except the model calls you explicitly configure.
The package ships the full pipeline — engine/, all 60 leaves' code, oracles, and prior
results — everything except the raw SEC documents themselves (fetch those via onboard or
source.py, per leaf) and, obviously, no model weights (those come from Ollama/DeepSeek/Gemini).
probity-bench demo # zero-config: replay a real wobble example, no install/network needed
probity-bench results # print the 2 summary tables from bundled scored.json
probity-bench list # every leaf + whether you've fetched its corpus
probity-bench run <leaf> # fetch (if needed) + benchmark one leaf with your configured modelsgit clone https://github.com/eikiyo/probity.git
cd probity
make setup # runs the test suite + regenerates results/RESULTS.md + this README's tables from diskThe benchmark itself has zero third-party dependencies — pure Python 3 stdlib, no network
call, no API key — so results/render.py and every engine module run on a bare interpreter. The
test suite needs pytest (fixtures and parametrize): pip install -e ".[dev]", or just
pip install pytest. Running the tests under unittest discover collects only a third of them
and exits 0, so don't.
To re-run a test yourself against live models (needs Ollama running
gemma3:1b locally + a DeepSeek API key — see .env.example):
cp .env.example .env && set -a && source .env && set +a
cd leaves/vesting_schedule # or any other leaf under leaves/
python3 source.py # fetch the real SEC documents into corpus/
python3 run.py # run the model ladder, N=20 each, writes scored.json
python3 ../../results/render.py # regenerate the tables with your fresh numbers60 tests, each item run 20x/item at temp 0.7 across a ladder of 12 models. Wobble (lower = better) is the run-to-run inconsistency rate, weighted by item count across every test that model ran; the per-category table below averages across all 12. Full per-test breakdown (all 60 tables): results/RESULTS.md.
| Model | Size | Tests covered | Wobble ↓ | Accuracy |
|---|---|---|---|---|
gemma3:1b |
1B, local | 60 |  |
 |
deepseek-v4-flash |
hosted, direct | 60 |  |
 |
deepseek-v4-pro |
hosted, direct | 60 |  |
|
gemma-4-31b-it |
31B, hosted (OR) | 60 |  |
 |
mistral-large-2512 |
hosted (OR) | 60 | |
 |
minimax-m2.5 |
hosted (OR) | 60 |  |
|
llama-3.3-70b |
70B, hosted (OR) | 60 | |
|
gemma3:1b-it-qat |
1B QAT, local | 60 |  |
 |
gemini-3-flash |
hosted (OR) | 60 | |
|
claude-haiku-4.5 |
hosted, direct API | 60 | |
|
gpt-oss-120b |
120B, hosted (OR) | 60 | |
|
gpt-5-mini |
hosted (OR) | 60 | |
|
| Category | Tests | Wobble ↓ (all models) | Accuracy (all models) |
|---|---|---|---|
| Priced equity rounds | 16 |  |
 |
| SAFEs & convertible notes | 12 | |
|
| Cap table math | 7 |  |
 |
| Investor rights & governance | 7 | |
 |
| Founder & employee vesting | 5 |  |
|
| Regulatory disclosures | 5 |  |
 |
| Off-market risk flags | 5 | |
 |
| Exit waterfalls | 3 |  |
 |
Not in general. Across 12 models measured at both legacy (0.7) and t01 (0.1) on the identical items, 6 wobbled less at t01 (0.1), 4 showed no difference, and 2 wobbled more at t01 (0.1) — 95% Tango intervals on the paired difference, counting only those that exclude zero.
The gain is concentrated in the weakest models: gemma3:1b improves by 19.5 points, while frontier models move by low single digits or not at all.
minimax-m2.5 got worse at t01 (0.1), by 3.2 points.
Accuracy is flat across both arms for every model: temperature moves consistency, not correctness.
Full table, per-model intervals and the parse-failure caveat: results/PAIRED_legacy_vs_t01.md.
Full per-item breakdown — including which clauses make each model wobble — in
results/RESULTS.md.
Most LLM benchmarks in niche domains are built from synthetic data with synthetic answers. That has a hidden flaw: if an AI writes both the question and the answer key, the answer key can be wrong in exactly the ways the model under test is wrong. Probity avoids this with a strict oracle layer:
- Source a real document that contains the ground truth in its own authoritative text — for example, a Certificate of Incorporation filed with the SEC that states, in legally precise language, whether its preferred stock is participating.
- A human separates the question from the answer. The model sees only the clause (the question). The validated label, plus the exact quote that proves it, is stored in a separate oracle file the model never sees. Items whose answer cannot be determined with confidence are excluded, not guessed.
- Run only the question through each model, N times, and score the majority answer against the validated label.
Synthetic instantiation is used only to multiply difficulty (varying numbers, off-market terms, ambiguous phrasing) on top of a real, human-validated seed — never as the sole source of truth.
Probity's full test backlog is a structured map of fundraising-reasoning capabilities
(engine/registry.json) — 67 atomic checks across priced equity, convertibles, cap-table math,
exit waterfalls, investor rights, founder equity, regulatory filings, and off-market risk flags.
60 are built and measured; the remaining 7 are scoped but not yet sourced. Each check is
built one at a time, to depth, against real sourced documents.
engine/ the model-agnostic core: clients, run harness, normalizer, reliability+accuracy scorers
leaves/ one folder per test, each with its real-document corpus, its separated oracle, and its runner
results/ the living benchmark tables + the paired temperature report
See the Quickstart above for the full clone → run → reproduce path.
Bug reports, new leaves, and sourcing improvements are welcome — see CONTRIBUTING.md. Security issues: see SECURITY.md, never a public issue.
MIT — see LICENSE.