Skip to content

docs(engine): say that the throughput rig is not reproducible - #72

Merged
donislawdev merged 1 commit into
masterfrom
docs/throughput-numbers-are-not-reproducible
Jul 28, 2026
Merged

docs(engine): say that the throughput rig is not reproducible#72
donislawdev merged 1 commit into
masterfrom
docs/throughput-numbers-are-not-reproducible

Conversation

@donislawdev

Copy link
Copy Markdown
Owner

Why

Re-measuring the same bare recv+send loop over the same loopback flood on the same machine, three times:

run pkt/s
1 14 488
2 30 189
3 15 004

A factor of two, unexplained. Ratios did not survive either: batched-vs-unbatched ran 1.23x to 3.65x, and "real NIC vs loopback" changed sign between runs (0.67x -> 1.32x). One run ordered a strategy doing strictly less work as slower, which is a reading of the noise floor rather than a result.

The figures in engine.py were written as if they were the tool's ceiling. They are internally consistent within one run, and that is all they are.

What this changes

  • New "HOW MUCH OF THIS TO TRUST" paragraph: a number from this rig is evidence only against another number taken in the SAME run. Do not quote the absolutes as a ceiling, do not derive a budget by dividing across runs, do not tune on a difference smaller than the spread.
  • Withdraws a claim that was never sound. The section divided a sniff-only reader's 92 944 pkt/s by the engine's ~14k and concluded "five sixths of the per-packet budget is spent after the read", naming the inject side as the suspect. Those two numbers come from different runs. What remains is the release heap growing under zero configured delay within one run (peak_queue 4080 -> 7020) - a direction, not a proportion.
  • Amends the RecvEx ADR in place rather than reversing it. Its argument was always the 1.4 packets per call - a direct observation of what the driver hands back, not a timing. The accompanying 1.29x / 1.40x now carry the caveat.
  • Records the send side honestly: batching was the fastest strategy in every run and on every path, by anything from 1.2x to 3.6x. Direction real, size not established.

The distinction that came out of this

Direct observations survived every repetition - a loss percentage against a control that lost 0.00%, a queue depth, scoped_seen, packets-per-call. Timings did not. That line is now drawn explicitly in the docstring and in PROJECT_NOTES rule 5, together with the method that was missing: repeat the whole run before believing a number, interleave strategies instead of running each in a block, carry a drift canary that makes the script refuse a conclusion rather than print a median, and report spread.

Verification

  • python -m pytest tests - 721 passed, 0 failed (elevated shell). Prose only, no behaviour change.
  • python smoke_gui.py - OK. check_notes.py - clean.

🤖 Generated with Claude Code

Re-measuring the same bare recv+send loop over the same loopback flood, same
machine, three times: 14 488 -> 30 189 -> 15 004 pkt/s. A factor of two, with
no explanation. Ratios moved too - batched-vs-unbatched ran 1.23x to 3.65x,
and "real NIC vs loopback" changed sign between runs (0.67x -> 1.32x). One run
even ordered a strategy doing strictly LESS work as slower, which is a reading
of the noise floor rather than a result.

The numbers in this docstring were presented as if they were the tool's
ceiling. They are internally consistent within one run and that is all.

- adds a HOW MUCH OF THIS TO TRUST paragraph: a number from this rig is
  evidence only against another number from the SAME run
- withdraws a derived claim that was never sound - dividing a sniff-only
  reader's 92 944 pkt/s by the engine's ~14k to conclude "five sixths of the
  budget is spent after the read". Different runs, so the arithmetic never
  held. What remains is the release heap growing under zero configured delay
  within one run: a direction, not a proportion.
- amends the RecvEx ADR in place instead of reversing it: its argument is the
  1.4 packets per call, a direct observation, not the 1.29x/1.40x timings
- records that SEND-side batching was the fastest strategy in every run and on
  every path, by anything from 1.2x to 3.6x. Direction real, size unknown.

No behaviour change: prose only.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@donislawdev
donislawdev merged commit cd9b681 into master Jul 28, 2026
8 checks passed
@donislawdev
donislawdev deleted the docs/throughput-numbers-are-not-reproducible branch July 28, 2026 20:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant