Make to-device the default ai-bot streaming mode

[lukemelia]

What

Makes to-device the default ai-bot streaming mode. The AI_BOT_STREAMING_MODE fallback flips from room-edits to to-device:

• mid-turn previews stream over the ephemeral to-device channel, and only one consolidated room edit lands per turn;
• room-edits (the legacy per-edit behavior) and off (no mid-turn previews) remain available as explicit AI_BOT_STREAMING_MODE overrides.

Why

A single active assistant conversation drives Synapse CPU largely through the volume of durable room events. A staging run comparing all three modes (fixed ~3,100-token response, 3 turns/mode) measured:

Mode	Room events/turn	To-device events/turn	Synapse CPU during load	Latency (median)
room-edits	142	0	0.14 cores (7.1%)	63.9s
off	2	0	0.02 cores (idle)	62.6s
to-device	2	~140	0.06 cores (3.0%)	61.8s

to-device cuts durable room events ~72× (142 → 2) while preserving mid-turn streaming (the previews just move to the to-device channel), at lower Synapse CPU than room-edits and with no latency regression. The deployed host client already stamps its originating device id, so previews stream for current clients; older clients that don't stamp it degrade gracefully to off-mode behavior (placeholder + final edit).

Changes

• responder.ts — flip the streamingMode fallback to to-device.
• main.ts — fix a stale env-var name in a comment (APP_BOXEL_STREAMING_MODE → AI_BOT_STREAMING_MODE).
• tests/responding-test.ts — the legacy tests exercise room-edit mechanics (thinking-message replacement, throttled edits, event splitting), so pin room-edits as the module baseline in beforeEach/afterEach; add a test asserting the unset default is now to-device.

Testing

• node tests/index.ts — green except the two AI Bot Locking tests, which require CI's Postgres role and pass there.
• pnpm lint:js / pnpm lint:types clean.

Deploy note

Making this the effective default in a deployed env requires that the ai-bot task definition not pin AI_BOT_STREAMING_MODE (an explicit env value overrides this code default). No infra env pin should be added.

🤖 Generated with Claude Code

[habdelra]

[Claude Code 🤖] Reviewed as a behavioral default-flip: does swapping the fallback to to-device change any reachable behavior, and does the test rebaseline preserve what the existing suite was actually exercising?

Bottom line: no blocking issues. The change is a clean fallback swap — the reachable mode set (off/room-edits/to-device) is unchanged, only the unset default moves — and the test rebaseline preserves the prior semantics of every existing test.

What lands right:

• The streamingMode refactor keeps all three modes reachable (mechanism in the inline on responder.ts). The to-device-without-target path degrades to off behavior (placeholder + one consolidated edit), and both halves of the "previews still stream for current clients" claim are verified against the host: it stamps the originating device id and consumes the to-device previews.
• Pinning room-edits in beforeEach and deleting the env var as the last step of afterEach exactly reproduces the pre-flip default for every non-mode-specific test — including the cleanup responder.finalize() in afterEach, which still runs under room-edits. So no existing test changes meaning; the pin is a faithful rebaseline, not a behavior edit.
• The main.ts comment fix is correct: APP_BOXEL_STREAMING_MODE appears nowhere in the tree, and AI_BOT_STREAMING_MODE is the actual variable the getter reads.

Operational caveat (not a code issue, out of scope for this diff): this becomes the effective default only where the ai-bot task definition does not pin AI_BOT_STREAMING_MODE. If the deployed task def currently pins a value, the change is a no-op there until the pin is removed. The deploy note already flags this — worth a confirm against the infra repo, since it's the single thing that decides whether this change does anything in a deployed env.

Minor (non-blocking): the new default streaming mode (no env var) is to-device test asserts roomEvents === 2 with the same no-preview-target setup as the existing ... without a target device falls back to `off` behavior test, so that half overlaps. The assertion unique to it — and the real guard on the flip — is telemetry.mode === 'to-device'. Both would catch a revert, so it's fine as-is; just noting the roomEvents check isn't the load-bearing part.

---

Generated by Claude Code

[habdelra]

[Claude Code 🤖] Confirmation — this is a fallback swap, not a change to the set of reachable modes (non-blocking).

The mode set is preserved. Before, {off, to-device} matched explicitly and everything else — including an explicit room-edits — fell to the room-edits fallback; now {off, room-edits} match explicitly and everything else — including to-device — falls to the to-device fallback. All three modes stay reachable; only the unset/unrecognized default moved.

The graceful-degradation path checks out end to end. shouldStreamMidTurn returns false for to-device when streamPreviewTarget is undefined, so a turn with no target lands only the placeholder + one consolidated room edit — the same shape as off. That's pinned by the `to-device` streaming mode without a target device falls back to `off` behavior test. On the client side, both halves of the "previews still stream for current clients" claim hold: the host stamps the originating device id (matrix-service.ts, originatingDeviceId = this.client.getDeviceId() → APP_BOXEL_ORIGINATING_DEVICE_ID_KEY) and consumes the previews (matrix-service.ts onToDeviceEvent → onResponseStreamPreview fan-out to the room resources).

Nothing to change here — this documents why the flip is safe. The one thing that would silently neutralize it lives outside the diff (an env pin on the task definition); see the review body.

---

Generated by Claude Code