Skip to content

feat: narrow the driver filter, and let the views follow the target - #73

Merged
donislawdev merged 7 commits into
masterfrom
feat/narrow-the-driver-filter-to-the-target
Jul 29, 2026
Merged

feat: narrow the driver filter, and let the views follow the target#73
donislawdev merged 7 commits into
masterfrom
feat/narrow-the-driver-filter-to-the-target

Conversation

@donislawdev

@donislawdev donislawdev commented Jul 28, 2026

Copy link
Copy Markdown
Owner

Chunks 1 and 2 of the throughput work. It turned out to be a correctness fix, not
the speed option it was designed as
- see the acceptance table.

The problem

With a destination target set, the tool receives every packet on the machine and
re-injects almost all of it untouched. Measured against a real capture: 1944 packets
diverted, 0 of them even eligible to be impaired
. Every one cost a recv plus a send
for nothing. The WinDivert filter runs in the driver, so whatever can be pushed
into it never reaches this process at all.

Acceptance, and the part worth reading twice

A flood to the targeted destination plus a decoy flood outside it:

run narrowed seen scoped_seen drop_loss sent to target receiver got
wide, no impairment False 75 056 15 768 0 28 050 15 768
narrow, no impairment True 27 950 27 950 0 27 950 27 950
wide, --loss 100 False 88 943 21 540 21 540 28 050 0
narrow, --loss 100 True 27 900 27 900 27 900 27 900 0

Without narrowing the driver was overloaded by traffic the tool was never going to
touch and discarded 43% of the TARGETED traffic before the tool saw it - so the
session impaired less than it claimed and computed its percentages over the
survivors. Narrowed: nothing lost, seen equals scoped_seen. The driver-wait
warning fired in both wide runs (52 / 184 ms) and in neither narrowed one - the same
overload from the other side.

The regression worth fearing did not happen: with --loss 100 the narrowed run
dropped 27 900 of 27 900 and the receiver got nothing, exactly as the wide run did.

How it is kept safe

The only invariant that matters: the driver filter must be a SUPERSET of the
matcher
. Over-capture is free (decide() still filters); under-capture is a silent
regression - traffic the user asked to impair would never arrive, with every counter
reading healthy.

  • only POSITIVE terms compile; dropping negatives can only widen
  • one term with no closed description (glob, re:) voids the whole expression
  • both directions are emitted - _capture_loop reads the remote endpoint as the
    DESTINATION outbound and as the SOURCE inbound, so a fragment testing only
    DstAddr/DstPort would have silently stopped impairing everything coming back.
    Caught by tracing writes, not by reading names.
  • the result must COMPILE, asked of the driver's own parser
  • process targeting can never be narrowed: processId is "bad token for layer" at
    NETWORK, checked rather than assumed

_Term now carries the parser's conclusion (shape) so the compiler reads the parse
result instead of parsing the same text twice - convention 10's concern is drift, and
two readers of one syntax stay in step only until one is edited.

The mid-session trap

dst_ip/dst_port are live-appliable; a handle's filter is not. Accepting a
destination change during a narrowed session would leave the driver filtering by the
old expression while decide() judged by the new one. apply_settings refuses a
changed destination out loud and leaves re-applying the same value alone.

Verification

  • python -m pytest tests - 732 passed, 0 failed on an elevated shell (721 -> 732, eleven new guards).
  • python smoke_gui.py - OK. tools/ci_gui_render.py - OK in both languages.
  • Oracle checked before being trusted: WinDivertHelperEvalFilter agreed with 40
    real captured packets across 9 filters, including the address-dependent terms a blank
    address struct would have got wrong. Then every fragment was swept: 27 648 packet
    views over 16 expressions, zero superset violations
    .
  • Eleven mutants across the two chunks. Nine caught. Two green ON PURPOSE, and both
    are the point: emitting only a range's lower bound is a widening and must not fail,
    and filter_compiles' exception path is unreachable where pydivert exists.
  • A surviving mutant found a real hole: removing the compile check went green
    because nothing produced a fragment the driver would refuse. It exists - the grammar
    has a length limit, and a 100-port list hits it (50 compile, 100 do not). Without the
    check the handle would fail to open at START. Guarded now.
  • The oracle test needs pydivert and is skipped without it - half the CI matrix.

Also in here

A Polish label typo the render check surfaced (invisible in code), and a measurement
with no code attached: WinDivertSendEx rejects a batch containing a malformed packet
atomically (pSendLen=0, nothing sent, 120 bytes of good packets ahead of it
included). That removes the one blocker named against the batched-injector chunk.

Also on this branch (added after the first review pass)

  • A view preference, "Show only the targeted traffic" (off by default): the counter grid, the
    chart, the Connections table and the connections CSV narrow to what the targeting selected. Three
    counters never narrow - drop_overflow, drop_shutdown, drop_send count what the TOOL lost,
    including untargeted traffic - and the notes and the chart caption re-word themselves so a
    screenshot cannot be misread. The stats CSV keeps both totals (it is an append log); NDJSON and
    the reproduction report are untouched.
  • A guard for every gate that judges the remote endpoint, in BOTH directions. Mutation showed
    the rule was caught only by three CONNECTION-LOG tests - nothing about targeting, LAN mode or
    blocking noticed. FakePacket also had src_port == dst_port, which made the port half of any
    such assertion vacuous; it takes them separately now.
  • An accounting hole closed: a packet popped off the release heap and then found without a
    divert (STOP cleared it in between) vanished with no counter at all. It is charged to
    drop_shutdown now.
  • The batched injector was built, measured and REVERTED - engine end to end 1.00x (7 of 14
    pairs, median 1.00, range 0.93-1.05). Instrumenting the real engine explains why: mean 2.13
    packets per batched call
    and peak_queue 30 against a 20 000 limit, because the inject thread
    keeps up and the release heap never accumulates. The ADR in CHANGELOG-INTERNAL names the number
    to re-measure before anyone reopens it.

Final state: 746 passed, smoke_gui.py OK, real-Tk render OK in both languages.

🤖 Generated with Claude Code

donislawdev and others added 7 commits July 28, 2026 23:40
With a destination target set the tool captures everything and re-injects
almost all of it untouched: measured 1944 packets diverted with 0 impairable,
and 1632 with 8. Each one costs a recv plus a send for nothing. The WinDivert
filter runs IN THE DRIVER, so whatever can be pushed into it never reaches this
process at all.

windivert_fragment(matcher) emits a fragment or None. Every _Term now carries a
`shape` - what the parser concluded - so the compiler reads the parse result
instead of parsing the same text a second time (convention 10: a second reader
of this syntax drifts at the first edit). matches() is untouched.

The invariant is the design: the fragment must be a SUPERSET of the matcher.
Over-capture is free because decide() still filters; under-capture is a silent
regression. So only positives are compiled, one shapeless term voids the whole
expression, and no address-dependent term is emitted.

Caught while tracing writes rather than reading names: _capture_loop reads the
remote endpoint as the DESTINATION outbound and as the SOURCE inbound, so a
fragment testing only DstAddr/DstPort would have kept every inbound packet away
from the tool - impairing one direction, silently dropping the other, counters
healthy throughout. Both directions are emitted and a test pins it.

Verified against the driver's own evaluator, not my reading of it:
- EvalFilter fidelity first, on 40 real captured packets across 9 filters, zero
  disagreements, including the address-dependent terms (sniff-only handle)
- then every fragment compile-tested and swept: 27 648 packet views over 16
  expressions, zero superset violations

Five mutants: four caught, and the fifth green ON PURPOSE - emitting only a
range's lower bound is a widening, and the tests must not reject it.

The oracle test needs pydivert and is skipped without it, which is half the CI
matrix; a green Linux run has not checked the invariant.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Normally the tool receives every packet on the machine and re-injects almost
all of it untouched - measured 1944 packets diverted with 0 impairable. The
WinDivert filter runs in the driver, so folding the destination into it means
that traffic never arrives here at all.

- filters.narrowed_filter() + filters.filter_compiles() (lazy pydivert; False
  when it cannot be asked, because "cannot prove" must mean "keep the wide
  filter")
- one registry entry `narrow_filter` (BOOL, start_only, settings surface), so
  the CLI flag, the widget, validation and the config file all follow
- engine.start(..., narrow=), start-only for the same reason duration is: a
  handle's filter is fixed when it opens
- only on the REAL path: an injected divert never reads the filter string, so
  narrowing there would move a number in the report without moving a packet

The mid-session trap and its guard: dst_ip/dst_port are live-appliable while
the handle's filter is not. Accepting a destination change during a narrowed
session would leave the driver filtering by the old expression while decide()
judged by the new one - traffic the user just asked to impair would never
arrive, with every counter healthy. apply_settings refuses a CHANGED
destination and says so; re-applying the same value is untouched.

Reported rather than only done: session_info()["narrowed"] (so the repro report
carries it) and capture_narrowed in the NDJSON summary. With narrowing on,
`packets` no longer counts every packet on the machine, so two reports with the
same key describe two different worlds. The CLI also says at START whether it
took effect - asking for the throughput and silently getting the wide filter
would otherwise go unnoticed.

A mutant survived and found a real hole: removing the compile check went green,
because nothing produced a fragment the driver would refuse. It exists - the
grammar has a length limit, and a 100-port list hits it (50 ports compile, 100
do not). Without the check the handle would fail to open at START. Guarded now.

Six mutants, five caught, one green on purpose.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Accepted on a real capture, and it is a correctness result rather than the
speed one it was designed as. A flood to the targeted destination plus a decoy
flood outside it:

  wide    28 050 sent to the target -> 15 768 in scope, 15 768 delivered
  narrow  27 950 sent to the target -> 27 950 in scope, 27 950 delivered

Without narrowing the driver was overloaded by traffic the tool was never going
to touch and discarded 43% of the TARGETED traffic before the tool saw it: the
session impaired less than it claimed and computed its numbers over the
survivors. Narrowed, nothing was lost and seen equals scoped_seen. The
driver-wait warning fired in both wide runs (52 and 184 ms) and in neither
narrowed one - the same overload from the other side.

The regression worth fearing did not happen: with --loss 100 the narrowed run
dropped 27 900 of 27 900 and the receiver got nothing, exactly as the wide run
did. A narrowing that also stopped the impairing would have read as a win in
every counter.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… bad batch

The label read "Zawez" instead of "Zawez" with the proper diacritics - invisible
in the code, obvious the moment the Settings window was rendered on real Tk. The
existing render check opens the About window, not Settings, so a control added
there is not covered by it; this one was measured directly (winfo_reqwidth vs
winfo_width across the whole window, nothing clipped, 181/181 for the new
checkbox).

Also records a measurement that needed no code: WinDivertSendEx rejects a batch
containing a malformed packet ATOMICALLY. Called through the DLL directly
(pydivert's wrapper raises before pSendLen can be read), with a truncated entry
in the middle of four:

  four good packets            rc=1  pSendLen=240 of 240
  one truncated in the middle  rc=0  pSendLen=0   of 190   err=122

120 bytes of good packets sat before the bad one and none went out. A batched
injector can therefore charge the whole batch to drop_send on failure, with no
partial-send ambiguity - which removes the blocker named against chunk 5.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
_capture_loop reads the remote endpoint as the packet's DESTINATION when it is
outbound and as its SOURCE when it is inbound. Five features consume that
value: destination IP, destination port, LAN mode, block by IP, block by port.

That rule was effectively unguarded, and this is measured rather than
suspected. Mutating the inbound branch to read dst_addr - the tidy-up anyone
would make while cleaning that block - was caught by three tests, ALL of them
about the connection log. Not one test about targeting, LAN mode or blocking
noticed. The tool would have gone on impairing outbound traffic and quietly
stopped impairing everything coming back, with every counter healthy.

The new file is table-driven on the CONSUMERS rather than on one feature, the
same shape as GATES in test_core_properties: adding a sixth reader of the
remote endpoint and forgetting the inbound case now fails at once. A mirror
test keeps it honest, since a gate that simply said "yes" would satisfy "fires
in both directions".

The fixture was hiding an entire axis: FakePacket set src_port and dst_port to
the SAME value, so swapping the inbound branch's ports changed nothing anywhere
in the suite - the port half of the assertion looked sound and tested nothing.
FakePacket now takes src_port/dst_port separately, defaulting to port so every
existing test is unchanged. Three direction mutants are caught now: inbound
address, inbound ports, outbound ports.

Also recorded because it cost a run: is_local_ip counts the TEST-NET ranges as
LOCAL (they are not globally routable), so the obvious test address makes LAN
mode do nothing and reads as a bug in the gate.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A new Settings switch, "Show only the targeted traffic", off by default. With it
on the counter grid, the throughput chart, the Connections table and the
connections CSV cover only what the targeting selected. It changes what you SEE
- never what is captured, never what is impaired.

One decider, not five: App.scoped_stat() + App.SCOPED_TWIN is the single place
that knows which counters have a narrowed twin, so the preference cannot mean
one thing on one tab and something else on the next.

Engine side stays lock-free. bytes_in_scoped/bytes_out_scoped are bumped in
_log_delivered from the row's own sticky `scoped` flag - already on the row by
then, so the heap tuple and its three positional consumers are untouched. No
_slock, on the same argument the sibling sent/sent_in/sent_out counters rest on:
the inject thread is their only writer and an int rebind is atomic. Taking the
lock there measured a 5% regression when those siblings were added.

Three deliberate omissions, each with its own test, because "we decided not to"
is what gets tidied away by somebody making things consistent:

- drop_overflow / drop_shutdown / drop_send NEVER narrow. They count what this
  tool lost, including untargeted traffic, and hiding that is the convention-20
  failure itself. Their tooltips say so in both languages.
- the stats CSV does not follow the switch (append log) and gains both totals as
  columns instead
- NDJSON and the reproduction report are untouched: they already carry seen AND
  scoped_seen, so a saved run never depends on how the window was set

The notes above both tables re-word themselves on the tick rather than only at
build, and the chart caption names the scope too - a chart is what people
screenshot, so it has to be readable from the picture. A note reading "ALL
captured traffic" over narrowed numbers is the rule-5 lie this guards against.

Five mutants, all caught, including the two good-faith "let's be consistent"
edits: giving a tool-loss counter a scoped twin, and counting scoped bytes for
every flow.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Found while building the batched injector, which was then measured and
reverted (ADR in CHANGELOG-INTERNAL). The hole is older than that work and
outlives it.

The inject loop pops a packet off the release heap, then finds _divert gone
because STOP cleared it in between. The packet is no longer in _heap, so
stop()'s stranded sweep cannot see it either - it simply left the
seen/delivered/dropped balance with no counter at all. That balance is the one
thing keeping these numbers honest: every other way a packet can die has a
counter, and one that quietly leaves the arithmetic makes a session's loss
figure wrong in the direction that flatters the tool.

It is charged to drop_shutdown now, which is what it is: dropped BY the
shutdown, not lost in transit.

Two guards: the popped-after-teardown case itself, and the
seen == delivered + lost balance across an ordinary pass-through session, so
the invariant is pinned for the common path and not only the exotic one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@donislawdev donislawdev changed the title feat: narrow the driver filter to the destination (chunks 1-2) feat: narrow the driver filter, and let the views follow the target Jul 29, 2026
@donislawdev
donislawdev merged commit adc9aaf into master Jul 29, 2026
8 checks passed
@donislawdev
donislawdev deleted the feat/narrow-the-driver-filter-to-the-target branch July 29, 2026 11:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant