feat: narrow the driver filter, and let the views follow the target - #73
Merged
Merged
Conversation
With a destination target set the tool captures everything and re-injects almost all of it untouched: measured 1944 packets diverted with 0 impairable, and 1632 with 8. Each one costs a recv plus a send for nothing. The WinDivert filter runs IN THE DRIVER, so whatever can be pushed into it never reaches this process at all. windivert_fragment(matcher) emits a fragment or None. Every _Term now carries a `shape` - what the parser concluded - so the compiler reads the parse result instead of parsing the same text a second time (convention 10: a second reader of this syntax drifts at the first edit). matches() is untouched. The invariant is the design: the fragment must be a SUPERSET of the matcher. Over-capture is free because decide() still filters; under-capture is a silent regression. So only positives are compiled, one shapeless term voids the whole expression, and no address-dependent term is emitted. Caught while tracing writes rather than reading names: _capture_loop reads the remote endpoint as the DESTINATION outbound and as the SOURCE inbound, so a fragment testing only DstAddr/DstPort would have kept every inbound packet away from the tool - impairing one direction, silently dropping the other, counters healthy throughout. Both directions are emitted and a test pins it. Verified against the driver's own evaluator, not my reading of it: - EvalFilter fidelity first, on 40 real captured packets across 9 filters, zero disagreements, including the address-dependent terms (sniff-only handle) - then every fragment compile-tested and swept: 27 648 packet views over 16 expressions, zero superset violations Five mutants: four caught, and the fifth green ON PURPOSE - emitting only a range's lower bound is a widening, and the tests must not reject it. The oracle test needs pydivert and is skipped without it, which is half the CI matrix; a green Linux run has not checked the invariant. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Normally the tool receives every packet on the machine and re-injects almost all of it untouched - measured 1944 packets diverted with 0 impairable. The WinDivert filter runs in the driver, so folding the destination into it means that traffic never arrives here at all. - filters.narrowed_filter() + filters.filter_compiles() (lazy pydivert; False when it cannot be asked, because "cannot prove" must mean "keep the wide filter") - one registry entry `narrow_filter` (BOOL, start_only, settings surface), so the CLI flag, the widget, validation and the config file all follow - engine.start(..., narrow=), start-only for the same reason duration is: a handle's filter is fixed when it opens - only on the REAL path: an injected divert never reads the filter string, so narrowing there would move a number in the report without moving a packet The mid-session trap and its guard: dst_ip/dst_port are live-appliable while the handle's filter is not. Accepting a destination change during a narrowed session would leave the driver filtering by the old expression while decide() judged by the new one - traffic the user just asked to impair would never arrive, with every counter healthy. apply_settings refuses a CHANGED destination and says so; re-applying the same value is untouched. Reported rather than only done: session_info()["narrowed"] (so the repro report carries it) and capture_narrowed in the NDJSON summary. With narrowing on, `packets` no longer counts every packet on the machine, so two reports with the same key describe two different worlds. The CLI also says at START whether it took effect - asking for the throughput and silently getting the wide filter would otherwise go unnoticed. A mutant survived and found a real hole: removing the compile check went green, because nothing produced a fragment the driver would refuse. It exists - the grammar has a length limit, and a 100-port list hits it (50 ports compile, 100 do not). Without the check the handle would fail to open at START. Guarded now. Six mutants, five caught, one green on purpose. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Accepted on a real capture, and it is a correctness result rather than the speed one it was designed as. A flood to the targeted destination plus a decoy flood outside it: wide 28 050 sent to the target -> 15 768 in scope, 15 768 delivered narrow 27 950 sent to the target -> 27 950 in scope, 27 950 delivered Without narrowing the driver was overloaded by traffic the tool was never going to touch and discarded 43% of the TARGETED traffic before the tool saw it: the session impaired less than it claimed and computed its numbers over the survivors. Narrowed, nothing was lost and seen equals scoped_seen. The driver-wait warning fired in both wide runs (52 and 184 ms) and in neither narrowed one - the same overload from the other side. The regression worth fearing did not happen: with --loss 100 the narrowed run dropped 27 900 of 27 900 and the receiver got nothing, exactly as the wide run did. A narrowing that also stopped the impairing would have read as a win in every counter. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… bad batch The label read "Zawez" instead of "Zawez" with the proper diacritics - invisible in the code, obvious the moment the Settings window was rendered on real Tk. The existing render check opens the About window, not Settings, so a control added there is not covered by it; this one was measured directly (winfo_reqwidth vs winfo_width across the whole window, nothing clipped, 181/181 for the new checkbox). Also records a measurement that needed no code: WinDivertSendEx rejects a batch containing a malformed packet ATOMICALLY. Called through the DLL directly (pydivert's wrapper raises before pSendLen can be read), with a truncated entry in the middle of four: four good packets rc=1 pSendLen=240 of 240 one truncated in the middle rc=0 pSendLen=0 of 190 err=122 120 bytes of good packets sat before the bad one and none went out. A batched injector can therefore charge the whole batch to drop_send on failure, with no partial-send ambiguity - which removes the blocker named against chunk 5. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
_capture_loop reads the remote endpoint as the packet's DESTINATION when it is outbound and as its SOURCE when it is inbound. Five features consume that value: destination IP, destination port, LAN mode, block by IP, block by port. That rule was effectively unguarded, and this is measured rather than suspected. Mutating the inbound branch to read dst_addr - the tidy-up anyone would make while cleaning that block - was caught by three tests, ALL of them about the connection log. Not one test about targeting, LAN mode or blocking noticed. The tool would have gone on impairing outbound traffic and quietly stopped impairing everything coming back, with every counter healthy. The new file is table-driven on the CONSUMERS rather than on one feature, the same shape as GATES in test_core_properties: adding a sixth reader of the remote endpoint and forgetting the inbound case now fails at once. A mirror test keeps it honest, since a gate that simply said "yes" would satisfy "fires in both directions". The fixture was hiding an entire axis: FakePacket set src_port and dst_port to the SAME value, so swapping the inbound branch's ports changed nothing anywhere in the suite - the port half of the assertion looked sound and tested nothing. FakePacket now takes src_port/dst_port separately, defaulting to port so every existing test is unchanged. Three direction mutants are caught now: inbound address, inbound ports, outbound ports. Also recorded because it cost a run: is_local_ip counts the TEST-NET ranges as LOCAL (they are not globally routable), so the obvious test address makes LAN mode do nothing and reads as a bug in the gate. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A new Settings switch, "Show only the targeted traffic", off by default. With it on the counter grid, the throughput chart, the Connections table and the connections CSV cover only what the targeting selected. It changes what you SEE - never what is captured, never what is impaired. One decider, not five: App.scoped_stat() + App.SCOPED_TWIN is the single place that knows which counters have a narrowed twin, so the preference cannot mean one thing on one tab and something else on the next. Engine side stays lock-free. bytes_in_scoped/bytes_out_scoped are bumped in _log_delivered from the row's own sticky `scoped` flag - already on the row by then, so the heap tuple and its three positional consumers are untouched. No _slock, on the same argument the sibling sent/sent_in/sent_out counters rest on: the inject thread is their only writer and an int rebind is atomic. Taking the lock there measured a 5% regression when those siblings were added. Three deliberate omissions, each with its own test, because "we decided not to" is what gets tidied away by somebody making things consistent: - drop_overflow / drop_shutdown / drop_send NEVER narrow. They count what this tool lost, including untargeted traffic, and hiding that is the convention-20 failure itself. Their tooltips say so in both languages. - the stats CSV does not follow the switch (append log) and gains both totals as columns instead - NDJSON and the reproduction report are untouched: they already carry seen AND scoped_seen, so a saved run never depends on how the window was set The notes above both tables re-word themselves on the tick rather than only at build, and the chart caption names the scope too - a chart is what people screenshot, so it has to be readable from the picture. A note reading "ALL captured traffic" over narrowed numbers is the rule-5 lie this guards against. Five mutants, all caught, including the two good-faith "let's be consistent" edits: giving a tool-loss counter a scoped twin, and counting scoped bytes for every flow. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Found while building the batched injector, which was then measured and reverted (ADR in CHANGELOG-INTERNAL). The hole is older than that work and outlives it. The inject loop pops a packet off the release heap, then finds _divert gone because STOP cleared it in between. The packet is no longer in _heap, so stop()'s stranded sweep cannot see it either - it simply left the seen/delivered/dropped balance with no counter at all. That balance is the one thing keeping these numbers honest: every other way a packet can die has a counter, and one that quietly leaves the arithmetic makes a session's loss figure wrong in the direction that flatters the tool. It is charged to drop_shutdown now, which is what it is: dropped BY the shutdown, not lost in transit. Two guards: the popped-after-teardown case itself, and the seen == delivered + lost balance across an ordinary pass-through session, so the invariant is pinned for the common path and not only the exotic one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Chunks 1 and 2 of the throughput work. It turned out to be a correctness fix, not
the speed option it was designed as - see the acceptance table.
The problem
With a destination target set, the tool receives every packet on the machine and
re-injects almost all of it untouched. Measured against a real capture: 1944 packets
diverted, 0 of them even eligible to be impaired. Every one cost a recv plus a send
for nothing. The WinDivert filter runs in the driver, so whatever can be pushed
into it never reaches this process at all.
Acceptance, and the part worth reading twice
A flood to the targeted destination plus a decoy flood outside it:
seenscoped_seendrop_loss--loss 100--loss 100Without narrowing the driver was overloaded by traffic the tool was never going to
touch and discarded 43% of the TARGETED traffic before the tool saw it - so the
session impaired less than it claimed and computed its percentages over the
survivors. Narrowed: nothing lost,
seenequalsscoped_seen. The driver-waitwarning fired in both wide runs (52 / 184 ms) and in neither narrowed one - the same
overload from the other side.
The regression worth fearing did not happen: with
--loss 100the narrowed rundropped 27 900 of 27 900 and the receiver got nothing, exactly as the wide run did.
How it is kept safe
The only invariant that matters: the driver filter must be a SUPERSET of the
matcher. Over-capture is free (
decide()still filters); under-capture is a silentregression - traffic the user asked to impair would never arrive, with every counter
reading healthy.
re:) voids the whole expression_capture_loopreads the remote endpoint as theDESTINATION outbound and as the SOURCE inbound, so a fragment testing only
DstAddr/DstPortwould have silently stopped impairing everything coming back.Caught by tracing writes, not by reading names.
processIdis "bad token for layer" atNETWORK, checked rather than assumed
_Termnow carries the parser's conclusion (shape) so the compiler reads the parseresult instead of parsing the same text twice - convention 10's concern is drift, and
two readers of one syntax stay in step only until one is edited.
The mid-session trap
dst_ip/dst_portare live-appliable; a handle's filter is not. Accepting adestination change during a narrowed session would leave the driver filtering by the
old expression while
decide()judged by the new one.apply_settingsrefuses achanged destination out loud and leaves re-applying the same value alone.
Verification
python -m pytest tests- 732 passed, 0 failed on an elevated shell (721 -> 732, eleven new guards).python smoke_gui.py- OK.tools/ci_gui_render.py- OK in both languages.WinDivertHelperEvalFilteragreed with 40real captured packets across 9 filters, including the address-dependent terms a blank
address struct would have got wrong. Then every fragment was swept: 27 648 packet
views over 16 expressions, zero superset violations.
are the point: emitting only a range's lower bound is a widening and must not fail,
and
filter_compiles' exception path is unreachable where pydivert exists.because nothing produced a fragment the driver would refuse. It exists - the grammar
has a length limit, and a 100-port list hits it (50 compile, 100 do not). Without the
check the handle would fail to open at START. Guarded now.
pydivertand is skipped without it - half the CI matrix.Also in here
A Polish label typo the render check surfaced (invisible in code), and a measurement
with no code attached:
WinDivertSendExrejects a batch containing a malformed packetatomically (
pSendLen=0, nothing sent, 120 bytes of good packets ahead of itincluded). That removes the one blocker named against the batched-injector chunk.
Also on this branch (added after the first review pass)
chart, the Connections table and the connections CSV narrow to what the targeting selected. Three
counters never narrow -
drop_overflow,drop_shutdown,drop_sendcount what the TOOL lost,including untargeted traffic - and the notes and the chart caption re-word themselves so a
screenshot cannot be misread. The stats CSV keeps both totals (it is an append log); NDJSON and
the reproduction report are untouched.
the rule was caught only by three CONNECTION-LOG tests - nothing about targeting, LAN mode or
blocking noticed.
FakePacketalso hadsrc_port == dst_port, which made the port half of anysuch assertion vacuous; it takes them separately now.
divert (STOP cleared it in between) vanished with no counter at all. It is charged to
drop_shutdownnow.pairs, median 1.00, range 0.93-1.05). Instrumenting the real engine explains why: mean 2.13
packets per batched call and
peak_queue30 against a 20 000 limit, because the inject threadkeeps up and the release heap never accumulates. The ADR in CHANGELOG-INTERNAL names the number
to re-measure before anyone reopens it.
Final state: 746 passed,
smoke_gui.pyOK, real-Tk render OK in both languages.🤖 Generated with Claude Code