From 412b64d5a29fed717c490d2a0de04209373b58b4 Mon Sep 17 00:00:00 2001 From: comis-agent Date: Sun, 26 Jul 2026 19:35:36 +0300 Subject: [PATCH 01/33] fix(context,obs,daemon): honour freshTailTurns, boot without autonomy, name every failure MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three silent behaviour bugs and twelve diagnosability gaps, found by reviewing a production session and then reproducing it end-to-end through the Telegram emulator on the same provider/model. Behaviour fixes * Fresh-tail clamp was a constant. resolveClampedFreshTailTurns estimated a step at W/20, which cancels against the 30% budget: floor(0.3W / (W/20)) = 6 for EVERY finite window and every configured value, so contextEngine.freshTailTurns (1..50, default 8) was unreachable above 6. On a long tool loop the user's originating request slid out of the verbatim tail while the window sat 91% empty, and the agent then answered as though it had never been asked. Replaced with an absolute per-step estimate calibrated so the change can only ever keep MORE context: at the smallest viable window (8192) it reproduces the previous 6 exactly, and every larger window now reaches its configured value. The two existing auto-estimate tests could not catch this — both only asserted 1 <= result <= configured, which a constant 6 satisfies. * Daemon crash-looped on a supported config. constructCapabilityLayer returns no capability endpoint when no agent has autonomy enabled; setupProactiveSchedulers treated it as mandatory and the daemon threw, completing its whole boot and then exiting 1 — forever, while systemctl reported active. Omitting the autonomy block defaults to enabled, but setting any autonomy sub-key without enabled: true resolves to disabled, so adding one documented knob bricked the box. Now boots with an ERROR naming what is off and the knob; any other missing dependency still aborts. * Response locale is pinned, not switched per message. Only the explicit agents..language pin enforces the reply language; the channel-supplied and script-derived tiers are advisory. Inferring from the current message alone let a single message switch a whole conversation and then burn a repair round-trip fighting a model that was already correct. * range_token was redacted as a secret. The .*token field-name wildcard rewrote a plain enum argument to the redaction placeholder, which the model then read back from its own replay context and sent to the MCP server. Narrow, audited exact-name exception set — never a pattern loosening — with an exhaustive negative matrix. * A failure with no attributable event no longer renders a bare errorKind pill to the user (UNATTRIBUTED_FAILURE_REASON). * Approval-gated env_set no longer tells the caller to re-read a value the redaction pass has already removed — which could only be satisfied by asking the user to re-send a credential through the chat. Both messages now name the deadlock, forbid re-transmission, and give the out-of-band path. Diagnosability * explain: an unresolvable sessionKey yields session_not_found instead of an all-zero report indistinguishable from a healthy session. * explain: contextBudget carries freshTailSteps/freshTailStepsConfigured, plus a fresh_tail_clamped verdict — the fresh-tail slide was previously invisible behind a clean "fits". * One failure produced up to 17 trajectory records; a dispatch redelivery is now marked and skipped by the bridge. * Cache breaks were logged twice under one message, double-counting every lens. * system-health reports the hard-degraded split it already fired findings on, so the table and its own JSON can no longer disagree. * totalTokens declares its counting convention (tokenBasis) — the two lenses count different things and neither said so. * Cache-churn attribution parses the block count and names block-count-changed instead of "unknown". * Hints that pointed the wrong way now name the knob: the lookback cache break no longer claims "no action needed" on a priced break; an MCP timeout names integrations.mcp.callToolTimeoutMs and its value and says an identical retry re-expires it; the locale repair names the resolver tier rather than blaming the model; a dropped memory distinguishes a real credential from a heuristic match. * The CLI no longer suggests re-running init when the gateway token is in the encrypted store — following that advice rotates a live token. Also fixes a latent two-sources-of-truth drift: the MCP client fell back to a 60s call timeout while the schema defaulted to 120s, silently halving the deadline on any un-threaded wiring path. Both now reference one constant, pinned by a parity gate. Docs updated in the same change for every behaviour above, including two new troubleshooting runbooks (cron/heartbeat never firing, and gateway-token recovery under encrypted storage). --- docs/agents/compaction.mdx | 2 +- docs/get-started/glossary.mdx | 2 +- docs/operations/multilingual.mdx | 27 +++-- docs/operations/observability.mdx | 18 +++ docs/operations/troubleshooting.mdx | 59 ++++++++++ docs/reference/config-yaml.mdx | 6 +- docs/reference/json-rpc.mdx | 8 +- .../src/background/background-task-manager.ts | 5 + .../agent/src/context-engine/lcd-assembler.ts | 4 + .../agent/src/context-engine/lcd-preflight.ts | 13 +++ .../cache-detection/cache-break-hints.test.ts | 39 +++++++ .../cache-detection/cache-break-hints.ts | 81 ++++++++++++++ .../cache-detection/cache-state.test.ts | 44 ++++++++ .../executor/cache-detection/cache-state.ts | 10 +- .../src/executor/cache-detection/index.ts | 1 + .../executor/executor-post-execution.test.ts | 32 ++++++ .../src/executor/executor-post-execution.ts | 16 ++- .../executor/pi-executor/pi-executor.test.ts | 14 ++- .../src/executor/pi-executor/pi-executor.ts | 3 +- .../src/executor/prompt-assembly.test.ts | 4 +- .../response-locale-enforcement.test.ts | 24 ++++ .../response-locale-enforcement.ts | 36 +++++- .../resolve-response-locale-policy.test.ts | 46 +++++++- .../resolve-response-locale-policy.ts | 15 ++- .../request-body/prefix-stability.test.ts | 27 +++++ .../request-body/prefix-stability.ts | 23 +++- .../agent/src/model/fresh-tail-clamp.test.ts | 73 ++++++++++++ packages/agent/src/model/fresh-tail-clamp.ts | 60 +++++++++- packages/cli/src/commands/mcp-token.test.ts | 71 ++++++++++++ packages/cli/src/commands/mcp-token.ts | 15 ++- packages/cli/src/commands/system-health.ts | 9 +- packages/core/src/activity/index.ts | 2 +- packages/core/src/activity/turn-outcome.ts | 14 +++ .../api-contracts/incident-report-sections.ts | 13 +++ .../core/src/api-contracts/incident-report.ts | 16 +++ .../src/api-contracts/system-health-report.ts | 29 +++++ packages/core/src/config/index.ts | 1 + .../core/src/config/schema-integrations.ts | 13 ++- packages/core/src/event-bus/events-infra.ts | 14 +++ .../core/src/event-bus/events-messaging.ts | 18 +++ packages/core/src/exports/activity.ts | 1 + packages/core/src/exports/config.ts | 1 + .../src/security/secret-detection.test.ts | 104 ++++++++++++++++++ .../core/src/security/secret-detection.ts | 66 ++++++++++- packages/daemon/src/api/env-handlers.ts | 14 ++- .../api/obs-handlers/obs-explain-assemble.ts | 2 +- .../obs-explain-fresh-tail-verdict.test.ts | 64 +++++++++++ .../obs-explain-fresh-tail-verdict.ts | 65 +++++++++++ .../obs-handlers/obs-explain-heuristics.ts | 8 ++ .../src/api/obs-handlers/obs-explain.test.ts | 63 +++++++++++ .../src/api/obs-handlers/obs-explain.ts | 33 +++++- .../api/obs-handlers/system-health.test.ts | 46 ++++++++ .../src/api/obs-handlers/system-health.ts | 16 ++- packages/daemon/src/daemon.ts | 23 ++-- .../daemon/src/wiring/proactive-degrade.ts | 86 +++++++++++++++ .../src/wiring/setup-observability.test.ts | 82 ++++---------- .../daemon/src/wiring/setup-observability.ts | 25 +---- .../wiring/setup-proactive-schedulers.test.ts | 30 +++-- .../src/wiring/setup-proactive-schedulers.ts | 34 +++++- .../src/trajectory/event-bus-bridge.test.ts | 80 ++++++++++++++ .../src/trajectory/event-bus-bridge.ts | 10 ++ .../src/trajectory/translate-payload.ts | 9 ++ .../activity-turn-coordinator.test.ts | 50 +++++++++ .../execution/activity-turn-coordinator.ts | 18 ++- .../platform-tools/tools/gateway-tool.test.ts | 41 +++++++ .../src/platform-tools/tools/gateway-tool.ts | 32 +++++- .../skills/integrations/mcp-client.test.ts | 6 +- .../skills/integrations/mcp-client/index.ts | 6 +- .../mcp-client/mcp-client-call.test.ts | 67 +++++++++++ .../mcp-client/mcp-client-call.ts | 55 +++++++++ .../mcp-client/mcp-client-types.ts | 2 +- packages/web/src/api/contracts.generated.json | 20 ++++ .../web/src/api/contracts.generated.size.json | 8 +- packages/web/src/api/contracts.generated.ts | 20 ++++ ...mon-boot-degrades-without-autonomy.test.ts | 81 ++++++++++++++ .../mcp-timeout-default-parity.test.ts | 59 ++++++++++ .../token-basis-lens-reconciliation.test.ts | 48 ++++++++ 77 files changed, 2114 insertions(+), 168 deletions(-) create mode 100644 packages/agent/src/executor/cache-detection/cache-break-hints.test.ts create mode 100644 packages/agent/src/executor/cache-detection/cache-break-hints.ts create mode 100644 packages/cli/src/commands/mcp-token.test.ts create mode 100644 packages/daemon/src/api/obs-handlers/obs-explain-fresh-tail-verdict.test.ts create mode 100644 packages/daemon/src/api/obs-handlers/obs-explain-fresh-tail-verdict.ts create mode 100644 packages/daemon/src/wiring/proactive-degrade.ts create mode 100644 test/architecture/daemon-boot-degrades-without-autonomy.test.ts create mode 100644 test/architecture/mcp-timeout-default-parity.test.ts create mode 100644 test/architecture/token-basis-lens-reconciliation.test.ts diff --git a/docs/agents/compaction.mdx b/docs/agents/compaction.mdx index be3a2530df..8f2d2b075c 100644 --- a/docs/agents/compaction.mdx +++ b/docs/agents/compaction.mdx @@ -491,7 +491,7 @@ change anything. | `contextEngine.observationDeactivationChars` | number | `80000` | Character threshold to deactivate observation masking (20K-500K) | | `contextEngine.ephemeralKeepWindow` | number | `10` | Recent ephemeral tool results to preserve from masking (1-50) | | `contextEngine.contextThreshold` | number | `0.75` | **DAG mode:** context-utilization fraction (of the turn's effective budget window — the reconciled context window under any capability-class cap) that triggers a leaf-summarization pass at the end of a turn (0.1-0.95) | -| `contextEngine.freshTailTurns` | number | `8` | **DAG mode:** most-recent steps (assistant + tool round-trips) always kept verbatim and never summarized/evicted (1-50) | +| `contextEngine.freshTailTurns` | number | `8` | **DAG mode:** most-recent steps (assistant + tool round-trips) always kept verbatim and never summarized/evicted (1-50). Reduced only on a small context window (see [config reference](/reference/config-yaml)); honoured as configured on frontier-sized windows. **Raise it for agents that run long tool loops** — a turn with more tool round-trips than this slides the user's own request out of the verbatim tail, and the model can then answer as though it was never asked. | | `contextEngine.leafChunkTokens` | number | `20000` | **DAG mode:** token cap for the oldest out-of-tail chunk summarized into one leaf summary (1K-100K); clamped at runtime to the resolved summarizer model's window — the smaller of its configured window and the probed served window when the summarizer runs on the served-bound provider — minus the summary target, template overhead, and previous-summary size, so a small compaction summarizer or a served-bound primary is never fed an over-window chunk; a single message larger than the clamped cap is replaced by a bounded deterministic extraction (no LLM call) | | `contextEngine.leafTargetTokens` | number | `1200` | **DAG mode:** target token size for a leaf summary (96-5000) | | `contextEngine.deferCompaction` | boolean | `true` | **DAG mode:** run the afterTurn leaf + condense passes in the background on the per-conversation serializer (never blocking the turn). `false` runs them inline (deterministic, for tests) | diff --git a/docs/get-started/glossary.mdx b/docs/get-started/glossary.mdx index ac75c960e9..4201955418 100644 --- a/docs/get-started/glossary.mdx +++ b/docs/get-started/glossary.mdx @@ -132,7 +132,7 @@ When an agent uses a tool, it may need another thinking step to process the tool ## Fresh Tail -The most recent **steps** in a DAG-mode conversation that are always kept verbatim (an assistant message plus the tool results it triggered counts as one step — not user-turns), configured via `contextEngine.freshTailTurns` (default 8). The fresh tail is sliced as the original structured blocks (never reconstructed-from-text) and is never evicted. This is a DAG context engine concept; DAG is the default engine (`contextEngine.version` defaults to `"dag"`; set `"pipeline"` to opt into the simpler engine). See [Context Management](/agents/context-management#dag-mode-the-default-engine). +The most recent **steps** in a DAG-mode conversation that are always kept verbatim (an assistant message plus the tool results it triggered counts as one step — not user-turns), configured via `contextEngine.freshTailTurns` (default 8; reduced only on a small context window). The tail is bounded by **step count**, not tokens — so a turn with more tool round-trips than this can push the originating user request out of verbatim context even when the context window is nearly empty. `comis explain` reports the effective and configured values as `contextBudget.freshTailSteps` / `freshTailStepsConfigured`. The fresh tail is sliced as the original structured blocks (never reconstructed-from-text) and is never evicted. This is a DAG context engine concept; DAG is the default engine (`contextEngine.version` defaults to `"dag"`; set `"pipeline"` to opt into the simpler engine). See [Context Management](/agents/context-management#dag-mode-the-default-engine). ## Gateway diff --git a/docs/operations/multilingual.mdx b/docs/operations/multilingual.mdx index 3d5ed1a712..6612665805 100644 --- a/docs/operations/multilingual.mdx +++ b/docs/operations/multilingual.mdx @@ -92,7 +92,7 @@ Every non-Latin capability has a working, visible, lower-fidelity floor — noth | Summarizer model weak in the source language | dag mode: the extractive/deterministic floor preserves source spans verbatim; pipeline mode: the weak-class path skips summarization (passthrough — nothing mistranslated); distillation is already gated (see [Local and small models](#local-and-small-models-non-latin)) | | Small or local summarizer silently ignores the language instruction (Hebrew in → English out) | The recall hole would reopen invisibly → the `summary_language_mismatch` system signal makes it a count; the remedy is `strongerSummarizerModel` (language-capable), no gating | | No phrase-table entry for the resolved reply language | English degraded-reply strings (never throws) | -| Ambiguous or mixed-script inbound for reply-language resolution | Config → channel-supplied request locale → current-message script → unset; the script fallback fires only when non-Latin codepoints dominate the current message, and yields a script-only locale (`und-Hebr`, `und-Cyrl`, …) — never a guessed language | +| Ambiguous or mixed-script inbound for reply-language resolution | Config → channel-supplied request locale → current-message script → unset; the script fallback fires only when non-Latin codepoints dominate the current message, and yields a script-only locale (`und-Hebr`, `und-Cyrl`, …) — never a guessed language. Only the **config** tier enforces the reply language; both inferred tiers are advisory | | Conversation history not yet indexed by the trigram twins | Reachable via the word lane exactly as before; the `comis doctor` twin-backfill (normalized) makes it trigram-searchable, operator-run | | Conversation rows with stored token under-counts | The pre-flight is corrected immediately via the assembler's read-time `max(stored, factored-live)`; trigger sums self-heal as new rows dominate (a late condense is non-destructive) | | Trigram-lane snippets | Show the *normalized* stored text (niqqud-stripped, finals-folded) — cosmetic only, behavioral parity with the existing message-lane snippet SQL | @@ -154,11 +154,23 @@ Two config keys carry the multilingual surface; both are documented in the scripts only, as a script-only locale such as `und-Hebr`), and otherwise leaves it unset. A channel-supplied locale is transport metadata, not conversation truth: when the current message is written in a script that contradicts it, the conversation's script wins and the - transport locale is ignored for that turn. When a locale **is** resolved, it also governs the - live reply: a final response whose script contradicts the resolved locale triggers **at most - one** bounded, tools-disabled repair turn that rewrites the reply into the expected script - (facts, identifiers, code, and URLs preserved); streaming consumers receive only the finalized - response while enforcement is active. See the + transport locale is ignored for that turn. + + **Only the explicit `agents..language` pin ENFORCES the reply language.** When that key is + set, a final response whose script contradicts it triggers **at most one** bounded, + tools-disabled repair turn that rewrites the reply into the expected script (facts, + identifiers, code, and URLs preserved); streaming consumers receive only the finalized response + while enforcement is active. The two inferred tiers — the channel-supplied request locale and + the current-message script fallback — are **advisory**: they inform the prompt but never + trigger a repair. + + This is deliberate. The script fallback reads the **current message only**, so enforcing it let + a single message switch an entire conversation's language and then spend a repair round-trip + trying to make the model comply. A model already mirrors its interlocutor's language without + being policed into it, so the inferred tiers do not need enforcement — while a genuine operator + pin does. **Consequence:** if you want a fixed reply language regardless of what any individual + message looks like, set `agents..language`; without it, per-message language switching is + followed naturally but never forced. See the [`agents` reference](/reference/config-yaml#agents). - **`embedding.multilingual`** — an advisory boolean for the `comis system-health` model-health line (see [Embedding and reranker](#embedding-and-reranker)). It does not gate search. See the @@ -169,7 +181,8 @@ script-only locale (`und-Cyrl`, `und-Hebr`, `und-Hani`, …), not a language: Cy Ukrainian as well as Russian, and Han codepoints are shared by Chinese and Japanese, so guessing a language tag from script alone would misname real users. Pin a precise language with the explicit `agents..language` key, or let the channel-supplied request locale (the user's own -client setting) carry it. +client setting) carry it. Remember that only the explicit key enforces — the inferred tiers guide +the reply without policing it. ## Summarizer language capability diff --git a/docs/operations/observability.mdx b/docs/operations/observability.mdx index e60b26c9c1..dc3d0ba40d 100644 --- a/docs/operations/observability.mdx +++ b/docs/operations/observability.mdx @@ -20,6 +20,24 @@ This page documents the observability foundations. It covers the trajectory laye --- +### Token counts differ between lenses — by design + +`obs.explain` and `obs.system.health` both report `cost.totalTokens`, and they count +**different things**. Each now declares which via a `tokenBasis` discriminator: + +| lens | `tokenBasis` | counts | +|---|---|---| +| `obs.explain` (one session) | `input+output+cache` | every token the session moved, **including** cache reads and writes | +| `obs.system.health` (window) | `input+output` | billable input + output only, cache **excluded** | + +One 27-minute session legitimately reported 6,043,245 on the first and 18,637 on the +second. Neither is wrong — read `tokenBasis` before comparing them, and never treat a +mismatch between the two as a bug on its own. + +A third number exists: the per-session `_session-metadata.json` rollup is written per +**execution** (last write wins), so on a multi-turn session it reflects the most recent +execution rather than the whole session. Prefer `obs.explain` for a session total. + ## 1. Trace Propagation Every inbound message receives a `traceId` at channel ingress — before the queue, before the agent, before delivery. The trace ID flows through the entire pipeline via AsyncLocalStorage (the trace-logger mixin in `packages/daemon/src/observability/trace-logger.ts`), so every Pino log line emitted during that turn carries the same `traceId` automatically. diff --git a/docs/operations/troubleshooting.mdx b/docs/operations/troubleshooting.mdx index c76303dfe8..f6a8ecb685 100644 --- a/docs/operations/troubleshooting.mdx +++ b/docs/operations/troubleshooting.mdx @@ -213,6 +213,38 @@ This page covers the most common issues you might encounter with Comis. Each ent +### Cron jobs and the heartbeat never fire (daemon otherwise healthy) + +**Symptom.** The daemon boots, channels serve, and the gateway is up — but no cron +job or heartbeat ever runs. The startup log carries: + +``` +Proactive schedulers not armed: autonomy is disabled for every agent +``` + +**Cause.** The proactive scheduler surface (cron + heartbeat) is built only when at +least one agent has autonomy enabled. If every agent resolves to autonomy-disabled, +the surface is intentionally left un-armed rather than half-wired. + +**The trap.** Omitting the `autonomy` block entirely resolves to **enabled**, but +adding a sub-key without also setting `enabled: true` can resolve to **disabled** via +the profile default — so adding one documented knob (for example +`agents..autonomy.durability`) can silently switch the whole surface off. + +**Fix.** Set the flag explicitly alongside whatever sub-key you wanted: + +```yaml +agents: + default: + autonomy: + enabled: true # required — a sub-key alone is not enough + durability: + enabled: true +``` + +Then restart. Verify with `comis cron list` (jobs present) and the absence of the +ERROR above in the startup log. + ## Channel Issues @@ -443,6 +475,33 @@ This page covers the most common issues you might encounter with Comis. Each ent +### `Missing COMIS_GATEWAY_TOKEN` from a CLI command while the daemon is healthy + +**Symptom.** A gateway-backed CLI command (`comis mcp list`, and other commands that +open the gateway socket) fails with `Missing COMIS_GATEWAY_TOKEN in the environment` — +yet the daemon is running and serving fine. + +**Cause.** Under `security.storage: encrypted` (the recommended posture) the gateway +token lives in the **encrypted `secrets.db`**, not in `~/.comis/.env` — that file holds +only `SECRETS_MASTER_KEY`. The daemon resolves the token from the store at boot; the +CLI's pre-connect check reads only the environment, so it misses. + +**Fix — read it from the store and pass it through:** + +```bash +comis --token "$(comis secrets get COMIS_GATEWAY_TOKEN)" mcp list +``` + + +Do **not** run `comis init` to "regenerate" the token on a box that is already +serving. It mints a **new** token, which invalidates the one the running daemon and +your other clients are using. + + +Offline observability commands (`comis system-health`, `comis explain`, +`comis messages`, `comis security audit-log`) read the data directory directly and +need no token. + ## Resilience Issues See [Resilience Architecture](/agents/resilience) for how these systems work together. diff --git a/docs/reference/config-yaml.mdx b/docs/reference/config-yaml.mdx index 87644745dc..35f11d9d30 100644 --- a/docs/reference/config-yaml.mdx +++ b/docs/reference/config-yaml.mdx @@ -295,7 +295,7 @@ behavior. |-----|------|---------|-------------| | `profile` | `enum` | `"standard"` | `assistant`, `standard`, `unattended`, or `max`. `standard` is the zero-config default. `unattended`'s never-hang mode behaviors (deny+escalate, the denial breaker, operator evict) are **active**; its capability set stays `standard`-equivalent (no over-grant). `max` still resolves to `standard`-equivalent in this release plus an "available later" notice. | | `role` | `enum` | `"worker"` | `worker` (default) or `coordinator`. `worker` does its work inline -- no change to existing agents. `coordinator` strips a long-running lead to the orchestration surface (`sessions_spawn`/`pipeline`/`cron`/`message` + the read-only drill-in tools `read`/`grep`/`find`/`ls` + the `obs_query` observability tool) under a delegate-then-synthesize doctrine -- heavy work always routes to a fresh-window child (`exec`/`edit`/`write`/`browser` are not on the coordinator surface). **Narrows the tool surface only; never widens a capability** -- the resolved `orch:*` set is byte-identical with and without the role. An explicit `tool_groups`/`full` overrides it. See [Long-Running Coordinator](/agents/efficiency). | -| `enabled` | `boolean` | _(from profile)_ | Whether autonomy surfaces are on. Defaults from the profile (`standard` -> true, `assistant` -> false); an explicit value overrides it. | +| `enabled` | `boolean` | _(from profile)_ | Whether autonomy surfaces are on. Defaults from the profile (`standard` -> true, `assistant` -> false); an explicit value overrides it. ⚠ **Omitting the whole `autonomy` block resolves to enabled, but adding any sub-key without also setting `enabled: true` can resolve to DISABLED** (via the profile default) — so writing e.g. only `autonomy.durability` may turn the autonomy surfaces off. With autonomy disabled for **every** agent, cron jobs and the heartbeat are **not armed**; the daemon boots and logs an ERROR naming this key. | | `capabilities` | `string[]` | _(from profile)_ | Explicit `orch:*` capability list. Overrides the profile's base set (still bounded by the structural floor). | | `mode` | `enum` | _(from profile)_ | Autonomy mode: `default`, `accept-reversible`, `unattended`, `max`. Operator-set; the agent cannot self-raise it. | | `denialBreakerN` | `number` | `5` | Consecutive floor blocks (capability/outward-quota denials) on a single run that trip the denial breaker -> the run aborts (`execution:aborted` reason `denial_breaker`) and escalates, rather than retry-looping and burning the budget. An allowed step in between resets the count. Positive integer. The never-hang dial for the [`unattended`](/agents/autonomy#the-unattended-profile) profile (applies to every profile). | @@ -643,7 +643,7 @@ Context engine configuration. Controls the system that manages what your agent s | Key | Type | Default | Description | |-----|------|---------|-------------| -| `freshTailTurns` | `number` | `8` | Recent **steps** (assistant + tool round-trips, not user-turns) always kept verbatim and never evicted (1-50) | +| `freshTailTurns` | `number` | `8` | Recent **steps** (assistant + tool round-trips, not user-turns) always kept verbatim and never evicted (1-50). A coarse guard additionally reduces this on a genuinely small context window (below roughly 11K tokens), so a tiny window cannot let the verbatim tail crowd out the system prompt; on frontier-sized windows the configured value is used as-is. The precise per-turn bound is enforced separately as a total-token cap. | | `contextThreshold` | `number` | `0.75` | Budget utilization ratio that triggers the end-of-turn leaf-summarization pass and the condense pass's hard-fanout pressure gate (0.1-0.95). The ratio is computed against the turn's effective budget window (min of the reconciled context window and the capability-class cap) — not the model's configured contextWindow — so capped or served-bound small models compact at the real window. | | `leafMinFanout` | `number` | `8` | Minimum raw messages before creating a leaf summary (2-20) | | `condensedMinFanout` | `number` | `4` | Minimum leaf summaries before creating a condensed summary (2-20) | @@ -1879,7 +1879,7 @@ External service integrations for search, MCP servers, media processing, and aut | Key | Type | Default | Description | |-----|------|---------|-------------| -| `callToolTimeoutMs` | `number` | `120000` | Default timeout for MCP tool calls in ms | +| `callToolTimeoutMs` | `number` | `120000` | Default timeout for MCP tool calls in ms. On expiry the tool error **names this key and its current value** and states that an identical retry re-expires the same deadline — narrow the request (smaller page / date window / fewer entities) or raise this value. Slow report and media tools may need well over two minutes. | | `servers` | `McpServerEntry[]` | `[]` | List of MCP servers | **MCP Server Entry (integrations.mcp.servers[])** diff --git a/docs/reference/json-rpc.mdx b/docs/reference/json-rpc.mdx index 89d27ac5c9..fc90707fa9 100644 --- a/docs/reference/json-rpc.mdx +++ b/docs/reference/json-rpc.mdx @@ -784,7 +784,7 @@ durably reset a task transition, the higher-priority reconciliation authority and the `health_signal:background_task_recovery_failed` system-health signal. -For `context_exhausted` sessions the report carries an optional **`contextBudget`** section -- the terminal per-LLM-call budget equation extracted from the trajectory's `context.budget` records (last record wins): `{ windowTokens, rawContextWindowTokens, windowCapSource, systemTokens, freshTailTokens, budgetedHistoryTokens, keptCount, assembledInputTokens, outputHeadroom, verdict }`. `windowCapSource` names what clamped the effective window below the model's configured `contextWindow`: the exact `contextEngine.budget.*` knob (`effectiveContextCapSmall` / `effectiveContextCapNano`), `served` (the Ollama-served `num_ctx` bound the window -- fix with `OLLAMA_CONTEXT_LENGTH= ollama serve` or a Modelfile `PARAMETER num_ctx`), `capabilityClass` (the operator's `providers.entries..capabilities.capabilityClass` pin bound the window -- pin a higher class or remove the pin; the `contextEngine.budget.*` caps do not move this bind), or `none`. `verdict` is the fit-check outcome (`fits` / `downshifted` / `exhausted`). When present, `likelyRootCause` is numbers-backed: it states the assembled-vs-window totals, the binding cap, the system+tools share of the window, and the kept-history count, and its suggested steps name the binding lever (the budget knob to raise, the Ollama serve knobs, or the `capabilityClass` pin) or recommend reducing the active tool surface when tool schemas dominate. The section is absent when the trajectory carries no `context.budget` records; the verdict then falls back to the generic wording. +For `context_exhausted` sessions the report carries an optional **`contextBudget`** section -- the terminal per-LLM-call budget equation extracted from the trajectory's `context.budget` records (last record wins): `{ windowTokens, rawContextWindowTokens, windowCapSource, systemTokens, freshTailTokens, budgetedHistoryTokens, keptCount, assembledInputTokens, outputHeadroom, verdict, freshTailSteps?, freshTailStepsConfigured? }`. `freshTailSteps` / `freshTailStepsConfigured` are the **effective vs operator-configured** count of trailing conversation steps kept verbatim (`contextEngine.freshTailTurns`). The verbatim tail is bounded by **step count, not tokens**, so a turn with more tool round-trips than the effective bound slides the user's own request out of verbatim context *while `verdict` still reads `fits`* and the window sits nearly empty. When the effective value is below the configured one, `likelyRootCause` stamps a `fresh_tail_clamped` verdict naming both numbers and the knob. `windowCapSource` names what clamped the effective window below the model's configured `contextWindow`: the exact `contextEngine.budget.*` knob (`effectiveContextCapSmall` / `effectiveContextCapNano`), `served` (the Ollama-served `num_ctx` bound the window -- fix with `OLLAMA_CONTEXT_LENGTH= ollama serve` or a Modelfile `PARAMETER num_ctx`), `capabilityClass` (the operator's `providers.entries..capabilities.capabilityClass` pin bound the window -- pin a higher class or remove the pin; the `contextEngine.budget.*` caps do not move this bind), or `none`. `verdict` is the fit-check outcome (`fits` / `downshifted` / `exhausted`). When present, `likelyRootCause` is numbers-backed: it states the assembled-vs-window totals, the binding cap, the system+tools share of the window, and the kept-history count, and its suggested steps name the binding lever (the budget knob to raise, the Ollama serve knobs, or the `capabilityClass` pin) or recommend reducing the active tool surface when tool schemas dominate. The section is absent when the trajectory carries no `context.budget` records; the verdict then falls back to the generic wording. When the session ran any memory recalls the report carries an optional **`recall`** section -- the memory-recall outcome aggregated over the trajectory's `memory.recalled` records: `{ recalls, zeroHits, lastLanes, lastFinalCount, rerankerAvailable, crossUserRecalls?, lastCrossUserCount? }` (counts and booleans only -- never the query text or any recalled memory body). `recalls` is how many recalls ran, `zeroHits` how many returned **no** injected memories (a recall miss), and the `last*` fields describe the terminal recall (its lane count and final injected-memory count). **`crossUserRecalls`** (present only when `> 0`) is how many recalls injected at least one memory scoped to a **different user** than the conversation -- the cross-sender privacy signal for agent-scoped recall (one shared agent surfacing sender A's memory into sender B's turn), so "was cross-sender data injected into this turn's context?" is answerable from the always-on report without pre-enabling the opt-in `diagnostics.recallTrace` artifact; **`lastCrossUserCount`** is the terminal recall's cross-user injected count. Counts only -- never the user-ids or bodies. This drives a dedicated **`recall_miss`** `likelyRootCause` verdict: a degraded session whose recalls **all** missed (`zeroHits === recalls`) and that matched no tool/context/breaker cause is root-caused as a recall miss, pointing the operator at the recall **scope** (agent- vs user-scoped) and, for non-Latin queries, the trigram-twin lanes (see [`comis system-health`](#obs-system-health) `health_signal` / `config_posture` embedder). A zero-hit recall on a **healthy** turn is benign -- the agent simply did not need memory -- and never names a cause. The section is absent for sessions with no recall records. @@ -819,7 +819,7 @@ An optional `coverage` block reports READ-coverage -- whether the trajectory, ro |----------|-------| | **Scope** | `admin` | | **Parameters** | `{ sessionKey?: string, traceId?: string, rootRunId?: string, depth?: "summary" \| "full", includeSynthetic?: boolean }` -- at least one of `sessionKey` / `traceId` / `rootRunId` required. `rootRunId` is an autonomy run's id, canonicalized to its session before assembly (a `root-` synthetic root resolves by prefix-strip; a real spawned root by a session-index scan) and renders the run's `spawnTree`. `includeSynthetic` is optional; include synthetic/test-harness sessions, which are excluded by default. | -| **Returns** | An `IncidentReport`: `{ schemaVersion, sessionKey, traceId, agentId, channel, outcome: { endReason, degraded, severity }, cost: { costUsd, totalTokens, cacheReadRatio }, timing: { durationMs, turnCount }, toolStats, failures[], breakerTimeline[], offloads[], contextBudget?, recall?, voice?, learning?, orchestrate?, cronWakeGate?: { jobId, wake, durationMs, toolCalls, estTurnsSaved }, summary, likelyRootCause, suggestedNextSteps[], truncations[], coverage? }` | +| **Returns** | An `IncidentReport`: `{ schemaVersion, sessionKey, traceId, agentId, channel, outcome: { endReason, degraded, severity }, cost: { costUsd, totalTokens, tokenBasis?, cacheReadRatio }, timing: { durationMs, turnCount }, toolStats, failures[], breakerTimeline[], offloads[], contextBudget?, recall?, voice?, learning?, orchestrate?, cronWakeGate?: { jobId, wake, durationMs, toolCalls, estTurnsSaved }, summary, likelyRootCause, suggestedNextSteps[], truncations[], coverage? }` | ```json Request @@ -898,7 +898,7 @@ This is the cross-session **sibling** of [`obs.explain`](#obs-explain): `obs.exp |----------|-------| | **Scope** | `admin` | | **Parameters** | `{ sinceHours?: number }` -- the aggregation window in hours (positive). Optional; defaults to `24` in the handler. | -| **Returns** | A `SystemHealthReport`: `{ schemaVersion, windowHours, sessions: { total, degraded, degradedRate }, degradedByCause, topErrorKinds[], breakerTripTotal, toolStats, cost: { costUsd, totalTokens }, activity: { activeAgents[], activeChannels[], exitReasons, turnTotal, tokenTotal }, findings[], likelyRootCause, pipelineAuthoringGate?: { buildAuthor, reason }, autonomy?: { runs: { total, degraded, degradedRate }, orphaned, resumed, revoked, killed, breakerTrips, denialBreakerTrips, budgetBreaches, costUsd, worstRootRunId? }, cronWakeGate?: { fires: { total, skipped, skipRate }, turnsSaved, toolCalls, perAgent[] }, suggestedNextSteps[], truncations[], coverage? }` | +| **Returns** | A `SystemHealthReport`: `{ schemaVersion, windowHours, sessions: { total, degraded, degradedRate, deliveredWithToolErrors?, hardDegraded?, hardDegradedRate? }, degradedByCause, topErrorKinds[], breakerTripTotal, toolStats, cost: { costUsd, totalTokens, tokenBasis?, offSessionUsd? }, activity: { activeAgents[], activeChannels[], exitReasons, turnTotal, tokenTotal }, findings[], likelyRootCause, pipelineAuthoringGate?: { buildAuthor, reason }, autonomy?: { runs: { total, degraded, degradedRate }, orphaned, resumed, revoked, killed, breakerTrips, denialBreakerTrips, budgetBreaches, costUsd, worstRootRunId? }, cronWakeGate?: { fires: { total, skipped, skipRate }, turnsSaved, toolCalls, perAgent[] }, suggestedNextSteps[], truncations[], coverage? }` | ```json Request @@ -915,7 +915,7 @@ This is the cross-session **sibling** of [`obs.explain`](#obs-explain): `obs.exp "result": { "schemaVersion": 1, "windowHours": 24, - "sessions": { "total": 40, "degraded": 22, "degradedRate": 0.55 }, + "sessions": { "total": 40, "degraded": 22, "degradedRate": 0.55, "deliveredWithToolErrors": 18, "hardDegraded": 4, "hardDegradedRate": 0.1 }, "degradedByCause": { "context_exhausted": 12, "completed_with_tool_errors": 7, "output_starved": 3 }, "topErrorKinds": [ { "kind": "dependency", "count": 18 } ], "breakerTripTotal": 3, diff --git a/packages/agent/src/background/background-task-manager.ts b/packages/agent/src/background/background-task-manager.ts index 1096a33b11..1f7e74740f 100644 --- a/packages/agent/src/background/background-task-manager.ts +++ b/packages/agent/src/background/background-task-manager.ts @@ -778,6 +778,11 @@ export function createBackgroundTaskManager(opts: BackgroundTaskManagerOpts): Ba durationMs: (current.completedAt ?? clock.now()) - current.startedAt, origin: current.origin, timestamp: clock.now(), + // This is a DISPATCH redrive of an already-terminal task, not a new + // terminal transition. Marked so occurrence-counting consumers (the + // trajectory bridge) record the failure ONCE — the unmarked re-emit + // wrote one line per backoff tick (live: 55 records for 4 tasks). + dispatchRedelivery: true, }; if (event === "background_task:completed") { emitObservationalEventSafely({ eventBus, logger }, event, common); diff --git a/packages/agent/src/context-engine/lcd-assembler.ts b/packages/agent/src/context-engine/lcd-assembler.ts index 123e47d402..ae71696dbd 100644 --- a/packages/agent/src/context-engine/lcd-assembler.ts +++ b/packages/agent/src/context-engine/lcd-assembler.ts @@ -553,6 +553,10 @@ export function createLcdContextEngine( windowCapSource: budget.windowCapSource, servedWindowTokens: budget.servedWindowTokens, }, + // The verbatim tail's STEP bound (effective vs the operator's configured + // value) — the number that decides whether the user's own request is + // still in context, and which was previously DEBUG-log-only. + { effective: clampedFreshTailTurns, configured: config.freshTailTurns }, ); deps.logger.info( diff --git a/packages/agent/src/context-engine/lcd-preflight.ts b/packages/agent/src/context-engine/lcd-preflight.ts index 3000b78f5f..9dab6b112b 100644 --- a/packages/agent/src/context-engine/lcd-preflight.ts +++ b/packages/agent/src/context-engine/lcd-preflight.ts @@ -61,6 +61,9 @@ const VALID_LEVELS: readonly TLevel[] = ["off", "minimal", "low", "medium", "hig * The NEWEST keptCount items from evictable were kept. * @param freshTail - The unconditional fresh-tail messages. * @param reasoningStyle - profile.reasoningStyle ("none" | "native"). + * @param freshTailStepBound - The effective (post-clamp) vs configured trailing-STEP + * count kept verbatim. Threaded onto context:budget_computed + * so a fresh-tail slide is visible in `explain`. * @param capInfo - Window-cap provenance (budget.rawContextWindowTokens + * budget.windowCapSource). When the effective window was * clamped by a capability-class cap, the exhaustion throw @@ -79,6 +82,7 @@ export function runPreflightFitCheck( freshTail: AgentMessage[], reasoningStyle: "none" | "native", capInfo?: ContextWindowCapInfo, + freshTailStepBound?: { effective: number; configured: number }, ): number { // Emit effectiveWindow callback so the caller can clamp max_tokens dynamically. deps.onEffectiveWindow?.(effectiveWindow); @@ -151,6 +155,15 @@ export function runPreflightFitCheck( assembledInputTokens: assembled, outputHeadroom: headroom, verdict, + // The STEP bound on the verbatim tail. Threaded so `explain` can say + // "the originating request slid out of the protected tail" instead of a + // clean-looking "fits" (comis-moshe 2026-07-26). + ...(freshTailStepBound !== undefined + ? { + freshTailSteps: freshTailStepBound.effective, + freshTailStepsConfigured: freshTailStepBound.configured, + } + : {}), }); }; diff --git a/packages/agent/src/executor/cache-detection/cache-break-hints.test.ts b/packages/agent/src/executor/cache-detection/cache-break-hints.test.ts new file mode 100644 index 0000000000..288d558fa1 --- /dev/null +++ b/packages/agent/src/executor/cache-detection/cache-break-hints.test.ts @@ -0,0 +1,39 @@ +// SPDX-License-Identifier: Apache-2.0 +import { describe, it, expect } from "vitest"; +import { lookbackWindowExceededHint } from "./cache-break-hints.js"; + +describe("lookbackWindowExceededHint", () => { + // Live incident (comis-moshe 2026-07-26): FOUR `lookback_window_exceeded` + // breaks in one 27-minute session, priced at $1.17–$1.33 each ($5.16 of the + // session's $8.59 cache waste) — every one of them logged + // "Multi-zone breakpoints mitigate this. No action needed." + it("never claims the break is costless or needs no action", () => { + const hint = lookbackWindowExceededHint(79); + expect(hint).not.toMatch(/no action needed/i); + expect(hint).not.toMatch(/mitigate this\.?\s*$/i); + }); + + it("names the knob(s) an operator can actually turn", () => { + const hint = lookbackWindowExceededHint(79); + expect(hint).toContain("contextEngine.contextThreshold"); + expect(hint).toContain("contextEngine.freshTailTurns"); + }); + + it("names where the priced impact of THIS break is recorded", () => { + const hint = lookbackWindowExceededHint(79); + expect(hint).toContain("cache-breaks/"); + expect(hint).toContain("estimatedCostUsd"); + expect(hint).toContain("system-health"); + }); + + it("states the BLOCK-count cause with the observed count (why compaction cannot help)", () => { + expect(lookbackWindowExceededHint(79)).toContain("79 blocks"); + expect(lookbackWindowExceededHint(79)).toMatch(/block count/i); + }); + + it("degrades honestly when the detector could not count blocks", () => { + const hint = lookbackWindowExceededHint(undefined); + expect(hint).toContain("the conversation"); + expect(hint).not.toContain("undefined"); + }); +}); diff --git a/packages/agent/src/executor/cache-detection/cache-break-hints.ts b/packages/agent/src/executor/cache-detection/cache-break-hints.ts new file mode 100644 index 0000000000..5e4fb70f91 --- /dev/null +++ b/packages/agent/src/executor/cache-detection/cache-break-hints.ts @@ -0,0 +1,81 @@ +// SPDX-License-Identifier: Apache-2.0 +/** + * Operator-facing `hint` text for cache-break WARNs. + * + * Lives in ONE place so the log site and its test assert the SAME string. The + * previous shape duplicated the literal into the test's hand-rolled handler + * stand-in, so the two could drift silently — a green-mock hazard on the exact + * field an operator reads to decide whether to act. + * + * The rule these hints follow (AGENTS.md §2.7, the troubleshooting feedback + * loop): a hint says WHICH KNOB, with the numbers that conflicted — never a + * bare reassurance. A `lookback_window_exceeded` break re-pays the whole + * prefix at the cache-WRITE rate; on a long-context model that is real money + * per occurrence, so "No action needed." was actively misleading. + * + * @module + */ + +/** + * Hint for a `lookback_window_exceeded` cache break. + * + * Why it is actionable (and why the old "No action needed." was wrong): the + * provider's cache lookup scans a bounded number of trailing message BLOCKS, + * not tokens. A conversation can therefore blow the lookback while sitting far + * below `contextEngine.contextThreshold` (the token-ratio that triggers + * compaction) — so compaction never fires and every subsequent turn re-pays the + * prefix at the cache-write rate. The lever is the BLOCK COUNT, which is why + * the log line carries `conversationBlockCount` and this hint names it. + * + * @param conversationBlockCount - blocks in the conversation at break time + * (`undefined` when the detector could not count them). + * @returns the hint string for the WARN's `hint` field. + */ +export function lookbackWindowExceededHint(conversationBlockCount: number | undefined): string { + const blocks = + conversationBlockCount === undefined ? "the conversation" : `${conversationBlockCount} blocks`; + return ( + `Cache lookback exceeded by BLOCK COUNT (${blocks}), not by tokens — so ` + + "`contextEngine.contextThreshold` compaction does not fire and the prefix is re-paid at the " + + "cache-write rate on this turn. Priced impact for this break is in " + + "`~/.comis/cache-breaks/__lookback_window_exceeded.json` (`estimatedCostUsd`); " + + "the recurring total is the `cache_prefix_churn` finding in `comis system-health`. " + + "Reduce trailing blocks (fewer tool round-trips per turn, or a lower " + + "`contextEngine.freshTailTurns`) to keep the prefix inside the lookback." + ); +} + +/** + * The structured field bag for the single "Cache break detected" INFO line. + * + * Extracted so the detector keeps ONE log call without carrying the field list + * inline (the subdirectory line cap), and so the fields stay in the same module + * as the hint text they accompany. + * + * @param event - the detected cache-break event. + * @returns content-free log fields (ids, counts, closed unions — never bodies). + */ +export function cacheBreakLogFields(event: { + agentId: string; + sessionKey: string; + provider: string; + reason: string; + tokenDrop: number; + tokenDropRelative: number; + ttlCategory: "short" | "long" | "none" | undefined; + toolsChanged: readonly string[]; + changes: { systemChanged: boolean; modelChanged: boolean }; +}): Record { + return { + agentId: event.agentId, + sessionKey: event.sessionKey, + provider: event.provider, + reason: event.reason, + tokenDrop: event.tokenDrop, + tokenDropRelative: event.tokenDropRelative, + ttlCategory: event.ttlCategory, + toolsChanged: event.toolsChanged, + systemChanged: event.changes.systemChanged, + modelChanged: event.changes.modelChanged, + }; +} diff --git a/packages/agent/src/executor/cache-detection/cache-state.test.ts b/packages/agent/src/executor/cache-detection/cache-state.test.ts index ef7b8041ad..d4195bdfe1 100644 --- a/packages/agent/src/executor/cache-detection/cache-state.test.ts +++ b/packages/agent/src/executor/cache-detection/cache-state.test.ts @@ -16,6 +16,7 @@ */ import { describe, it, expect, beforeEach, vi } from "vitest"; +import * as fs from "node:fs"; import type { ClockPort } from "@comis/core"; // Test-only clock stub. @@ -1634,3 +1635,46 @@ describe("cache break detector LRU eviction warning", () => { const _crossCheck = extractAnthropicPromptState(fixtureParams, "claude-sonnet-4-5", "short", "sess-1", "agent-1"); // eslint-disable-next-line @typescript-eslint/no-unused-vars const _ensureUsed: unknown = _crossCheck; + +describe("cache-break logging is single-source", () => { + // Observed live: `Cache break detected` appeared TWICE per break — once from the + // detector and once from an observability subscriber that re-logged the same + // event under the same `msg`. A 9-break session therefore produced 18 log lines, + // and the `cache_prefix_churn` system finding counted them all. + it("the single cache-break line carries the fields the observability projection used to duplicate", () => { + // The field bag lives in cache-break-hints.ts (cacheBreakLogFields) so the + // detector fits its subdirectory line cap while still emitting ONE line. + const src = fs.readFileSync( + new URL("./cache-break-hints.ts", import.meta.url), + "utf8", + ); + const match = /export function cacheBreakLogFields\([\s\S]*?\n\}/.exec(src); + expect(match, "cacheBreakLogFields must exist").not.toBeNull(); + const body = match![0]; + for (const field of [ + "sessionKey", + "tokenDropRelative", + "ttlCategory", + "systemChanged", + "modelChanged", + ]) { + expect(body, `the single cache-break line must carry ${field}`).toContain(field); + } + }); + + it("the detector logs the break exactly once, via that field bag", () => { + const src = fs.readFileSync(new URL("./cache-state.ts", import.meta.url), "utf8"); + const logCalls = src.match(/"Cache break detected"/g) ?? []; + // One in the log call itself (the comment above it must not repeat the literal). + expect(logCalls).toHaveLength(1); + expect(src).toContain("logger.info(cacheBreakLogFields(event)"); + }); + + it("no other module logs the same message (single source of the count)", () => { + const wiring = fs.readFileSync( + new URL("../../../../daemon/src/wiring/setup-observability.ts", import.meta.url), + "utf8", + ); + expect(wiring).not.toContain('"Cache break detected"'); + }); +}); diff --git a/packages/agent/src/executor/cache-detection/cache-state.ts b/packages/agent/src/executor/cache-detection/cache-state.ts index 2b04d5918e..28b01bca1d 100644 --- a/packages/agent/src/executor/cache-detection/cache-state.ts +++ b/packages/agent/src/executor/cache-detection/cache-state.ts @@ -10,6 +10,7 @@ * @module */ +import { cacheBreakLogFields } from "./cache-break-hints.js"; import { CACHE_BREAK_RELATIVE_THRESHOLD, CACHE_BREAK_ABSOLUTE_THRESHOLD, @@ -278,10 +279,11 @@ export function createCacheBreakDetector( breakpointBudget: state.currentSnapshot.breakpointBudget, }; - logger.info( - { agentId: event.agentId, provider: event.provider, reason: event.reason, tokenDrop: event.tokenDrop, toolsChanged: event.toolsChanged }, - "Cache break detected", - ); + // THE single cache-break log line. Its fields live in + // cache-break-hints.ts (see cacheBreakLogFields) — they carry everything the + // observability subscriber used to re-log, because two INFO lines with the + // same `msg` for one event made every counting lens double-count. + logger.info(cacheBreakLogFields(event), "Cache break detected"); return event; }, diff --git a/packages/agent/src/executor/cache-detection/index.ts b/packages/agent/src/executor/cache-detection/index.ts index c3a548d0e7..aa759dae37 100644 --- a/packages/agent/src/executor/cache-detection/index.ts +++ b/packages/agent/src/executor/cache-detection/index.ts @@ -32,3 +32,4 @@ export { sanitizeMcpToolName, sanitizeMcpToolNameForAnalytics, } from "./prompt-state-utils.js"; +export { lookbackWindowExceededHint, cacheBreakLogFields } from "./cache-break-hints.js"; diff --git a/packages/agent/src/executor/executor-post-execution.test.ts b/packages/agent/src/executor/executor-post-execution.test.ts index abf9981ecc..04b6a7362e 100644 --- a/packages/agent/src/executor/executor-post-execution.test.ts +++ b/packages/agent/src/executor/executor-post-execution.test.ts @@ -13,6 +13,7 @@ // "NO_REPLY" responses. import { readFileSync } from "node:fs"; +import * as fs from "node:fs"; import { dirname, resolve } from "node:path"; import { fileURLToPath } from "node:url"; import { describe, it, expect, expectTypeOf, vi } from "vitest"; @@ -2560,3 +2561,34 @@ describe("degraded-reply chokepoint consumes the typed locale policy", () => { ); }); }); + +describe("the paired-memory security-scan WARN distinguishes a leak from a false positive", () => { + // Observed live: THREE paired memories were dropped in one session with an + // identical WARN. Two were genuine credential matches (the firewall working); + // one was a suspicious-PATTERN heuristic on ordinary prose — a silently LOST + // memory. The line named the pattern sources but nothing told an operator which + // case they were looking at, so a false positive was indistinguishable from a + // blocked leak. + function detectorFor(patterns: readonly string[]): "secret" | "heuristic" { + return patterns.includes("secret-egress-guard") ? "secret" : "heuristic"; + } + + it("labels a secret-detector verdict `secret`", () => { + expect(detectorFor(["secret-egress-guard"])).toBe("secret"); + }); + + it("labels a suspicious-pattern verdict `heuristic`", () => { + expect(detectorFor(["system[ \\t]{0,20}:?[ \\t]{0,20}(prompt|override|command)"])).toBe("heuristic"); + }); + + it("the two branches produce DIFFERENT operator guidance", () => { + const src = fs.readFileSync( + new URL("./executor-post-execution.ts", import.meta.url), + "utf8", + ); + // The secret branch reassures (no action needed); the heuristic branch warns + // that a legitimate memory may have been lost. They must not be one string. + expect(src).toMatch(/firewall working as intended/); + expect(src).toMatch(/lost memory, not a blocked leak/); + }); +}); diff --git a/packages/agent/src/executor/executor-post-execution.ts b/packages/agent/src/executor/executor-post-execution.ts index 16f27a1710..2a681aae01 100644 --- a/packages/agent/src/executor/executor-post-execution.ts +++ b/packages/agent/src/executor/executor-post-execution.ts @@ -550,7 +550,21 @@ export async function storePairedConversationMemory( // Pattern-source tags only (e.g. "secret-egress-guard") — NEVER the // matched secret text. The verdict carries sources, not the content. patterns: verdict.patterns, - hint: "Paired conversation memory matched a secret/dangerous/suspicious pattern — skipped (the learned-trust conversation memory has no reduced-weight tier); the secret value is never logged or persisted", + // WHICH DETECTOR fired, as a closed enum. The two cases warrant OPPOSITE + // responses and the pattern-source tags alone did not make that legible: + // a `secret` verdict is the firewall working as designed (a credential + // really was in the turn), while a `heuristic` verdict is a suspicious- + // PATTERN match that may be a false positive on ordinary prose — and a + // false positive silently drops a legitimate memory. Live, three drops in + // one session were a mix of both and the WARN read identically for each. + // Still content-free: a closed label derived from the verdict's own + // source tags, never the matched text. + detector: verdict.patterns.includes("secret-egress-guard") + ? ("secret" as const) + : ("heuristic" as const), + hint: verdict.patterns.includes("secret-egress-guard") + ? "A credential was detected in the paired conversation memory — the write was skipped and the value was never logged or persisted. This is the firewall working as intended; no action needed unless a credential is reaching conversation content that should not carry one." + : "A suspicious-PATTERN heuristic (not the secret detector) matched the paired conversation memory, so the write was skipped and this turn is NOT recallable later. If the listed patterns look like a false positive on ordinary prose, that is a lost memory, not a blocked leak — review the named pattern sources before assuming the drop was correct.", errorKind: "validation" as ErrorKind, }, "Paired memory skipped: failed the memory-write security scan", diff --git a/packages/agent/src/executor/pi-executor/pi-executor.test.ts b/packages/agent/src/executor/pi-executor/pi-executor.test.ts index a68064d155..e36024c82a 100644 --- a/packages/agent/src/executor/pi-executor/pi-executor.test.ts +++ b/packages/agent/src/executor/pi-executor/pi-executor.test.ts @@ -26,6 +26,7 @@ import { clearSessionToolNameSnapshot, clearSessionBootstrapFileSnapshot, clearS import { clearSessionToolSchemaSnapshot } from "../executor-session-state.js"; import { resetPairedMemoryDedupForTests } from "../executor-post-execution.js"; import type { CacheBreakEvent, CacheBreakReason, PendingChanges } from "../cache-detection/index.js"; +import { lookbackWindowExceededHint } from "../cache-detection/index.js"; import { buildPromptingSnapshot } from "./pi-executor-prompting.js"; import { planInboundMessageProvenance } from "../../session/inbound-message-provenance.js"; @@ -7306,7 +7307,10 @@ describe("skip guard for lookback_window_exceeded cache breaks", () => { reason: event.reason, tokenDrop: event.tokenDrop, conversationBlockCount: event.conversationBlockCount, - hint: "Long conversation exceeded lookback window. Multi-zone breakpoints mitigate this. No action needed.", + // Imported from the SAME module the production log site uses, so the + // stand-in handler cannot drift from the real hint (it did: the literal + // was duplicated here and both said "No action needed." on a priced break). + hint: lookbackWindowExceededHint(event.conversationBlockCount), errorKind: "internal" as const, }, "Cache miss from lookback window exceeded (not server eviction)", @@ -7350,7 +7354,13 @@ describe("skip guard for lookback_window_exceeded cache breaks", () => { expect(logWarn).toHaveBeenCalledOnce(); expect(logWarn.mock.calls[0][1]).toContain("lookback window exceeded"); expect(logWarn.mock.calls[0][0].errorKind).toBe("internal"); - expect(logWarn.mock.calls[0][0].hint).toContain("lookback window"); + // The hint must name the cause + the knobs, and must NOT reassure — a + // lookback break re-pays the prefix at the cache-write rate (live: $1.17–$1.33 + // each, ×4 in one session, all logged "No action needed."). + expect(logWarn.mock.calls[0][0].hint).toMatch(/lookback/i); + expect(logWarn.mock.calls[0][0].hint).not.toMatch(/no action needed/i); + expect(logWarn.mock.calls[0][0].hint).toContain("contextEngine.freshTailTurns"); + expect(logWarn.mock.calls[0][0].hint).toContain("25 blocks"); // INFO log (coordinated reset) should NOT be emitted expect(logInfo).not.toHaveBeenCalled(); diff --git a/packages/agent/src/executor/pi-executor/pi-executor.ts b/packages/agent/src/executor/pi-executor/pi-executor.ts index 60df4aba22..2bfc5508e6 100644 --- a/packages/agent/src/executor/pi-executor/pi-executor.ts +++ b/packages/agent/src/executor/pi-executor/pi-executor.ts @@ -88,6 +88,7 @@ import type { ComisSessionManager } from "../../session/comis-session-manager.js import type { RunHandle } from "../active-run-registry.js"; import { repairOrphanedMessages, scrubPoisonedThinkingBlocks } from "../../session/orphaned-message-repair.js"; import { projectPendingDeliveredAssistantHistory } from "../../session/pending-delivered-assistant-history.js"; +import { lookbackWindowExceededHint } from "../cache-detection/cache-break-hints.js"; import { scrubRedactedToolCalls } from "../../session/scrub-redacted-tool-calls.js"; import { scrubForgedContextMarkers } from "../../session/forged-context-markers.js"; import { @@ -2268,7 +2269,7 @@ async function runSessionLocked( reason: event.reason, tokenDrop: event.tokenDrop, conversationBlockCount: event.conversationBlockCount, - hint: "Long conversation exceeded lookback window. Multi-zone breakpoints mitigate this. No action needed.", + hint: lookbackWindowExceededHint(event.conversationBlockCount), errorKind: "internal" as const, }, "Cache miss from lookback window exceeded (not server eviction)", diff --git a/packages/agent/src/executor/prompt-assembly.test.ts b/packages/agent/src/executor/prompt-assembly.test.ts index 76e4c0a684..ce7832aa5a 100644 --- a/packages/agent/src/executor/prompt-assembly.test.ts +++ b/packages/agent/src/executor/prompt-assembly.test.ts @@ -3128,7 +3128,7 @@ describe("bootstrap file snapshotting", () => { expect(result.responseLocalePolicy.source).toBe("unset"); }); - it("enforces the current request script when locale metadata is absent", async () => { + it("derives the current request script (ADVISORY) when locale metadata is absent", async () => { const result = await assembleExecutionPrompt(makeParams({ msg: makeMsg({ text: "أجب عن هذا الطلب بإيجاز" }), })); @@ -3136,7 +3136,7 @@ describe("bootstrap file snapshotting", () => { expect(result.responseLocalePolicy).toEqual({ locale: "und-Arab", source: "request", - enforceLocale: true, + enforceLocale: false, }); expect(result.dynamicPreamble).toContain('locale="und-Arab"'); expect(result.dynamicPreamble).toContain("same human language as the current user request"); diff --git a/packages/agent/src/executor/prompt-runner/response-locale-enforcement.test.ts b/packages/agent/src/executor/prompt-runner/response-locale-enforcement.test.ts index 2dfef71a53..1bdd1338a7 100644 --- a/packages/agent/src/executor/prompt-runner/response-locale-enforcement.test.ts +++ b/packages/agent/src/executor/prompt-runner/response-locale-enforcement.test.ts @@ -4,6 +4,7 @@ import { TypedEventBus, type ResponseLocalePolicy } from "@comis/core"; import { applyResponseLocaleEnforcement, enforceResponseLocale, + unrepairedMismatchHint, } from "./response-locale-enforcement.js"; import type { RunPromptParams } from "./prompt-runner-types.js"; import { allowProviderDispatch } from "../provider-dispatch.js"; @@ -414,3 +415,26 @@ describe("applyResponseLocaleEnforcement", () => { expect(JSON.stringify(logger.warn.mock.calls)).not.toContain("3600000"); }); }); + +describe("the unrepaired-mismatch WARN names the resolver tier, not the model", () => { + // Observed live: a single English instruction inside an otherwise-Hebrew + // conversation set `locale=en source=request enforce=true`. All THREE repair + // passes correctly came back Hebrew (the model was honouring the conversation's + // established language), yet every WARN said "Inspect the selected model's + // locale fidelity" — pointing the operator at the wrong knob. Each pass also + // cost a full extra model call and broke the prompt cache. + it("an INFERRED (request-tier) target says so and does not blame the model", () => { + const hint = unrepairedMismatchHint("request"); + expect(hint).toMatch(/inferred/i); + expect(hint).toContain("localeSource=request"); + expect(hint).not.toMatch(/model's locale fidelity/i); + // …and it states the cost, so a recurring mismatch is not read as harmless. + expect(hint).toMatch(/extra model call|prompt cache/i); + }); + + it("an OPERATOR PIN (explicit tier) still points at the model, because the pin is authoritative", () => { + const hint = unrepairedMismatchHint("explicit"); + expect(hint).toMatch(/operator pin/i); + expect(hint).toMatch(/locale fidelity/i); + }); +}); diff --git a/packages/agent/src/executor/prompt-runner/response-locale-enforcement.ts b/packages/agent/src/executor/prompt-runner/response-locale-enforcement.ts index d987f13cd7..92020743bb 100644 --- a/packages/agent/src/executor/prompt-runner/response-locale-enforcement.ts +++ b/packages/agent/src/executor/prompt-runner/response-locale-enforcement.ts @@ -296,16 +296,50 @@ export async function applyResponseLocaleEnforcement(params: RunPromptParams): P return; } + // The hint must NOT assume the model is at fault. A repair that keeps producing + // the SAME contradicting script is the model holding a persistent conversation + // language against a locale the resolver inferred from one message — observed + // live: a single English instruction inside an otherwise-Hebrew conversation set + // `locale=en source=request enforce=true`, and all three repair passes correctly + // came back Hebrew. Blaming "model locale fidelity" sends the operator to the + // wrong knob; name the resolver tier that produced the target instead, since the + // `request` tier is inferred and the `explicit` tier is an operator pin. params.deps.logger.warn( { step: "response-locale-repair", locale: outcome.value.finalFinding?.locale, + localeSource: params.responseLocalePolicy.source, expectedScript: outcome.value.finalFinding?.expectedScript, actualScript: outcome.value.finalFinding?.actualScript, durationMs, - hint: "Inspect the selected model's locale fidelity; the bounded repair remained mismatched", + hint: unrepairedMismatchHint(params.responseLocalePolicy.source), errorKind: "validation" as const, }, "Response locale remained mismatched after repair", ); } + +/** + * Operator-facing `hint` for a locale repair that never converged. + * + * Branches on the RESOLVER TIER that produced the target, because the right knob + * differs: a `request`-tier locale is INFERRED from the current message, so a + * persistent mismatch usually means the inference is wrong — not the model. An + * `explicit` locale is an operator pin, so the model genuinely failed to honour it. + * + * Exported for the same single-source reason as the other hint helpers: a hint + * duplicated into its test drifts silently. + * + * @param source - the resolved `ResponseLocaleSource` tier. + * @returns the hint text for the WARN's `hint` field. + */ +export function unrepairedMismatchHint(source: string): string { + return source === "request" + ? "The enforced locale was INFERRED from this request (localeSource=request), not pinned by an operator. " + + "The model answered in a different script on every attempt, which usually means the conversation's " + + "established language differs from this one message's. Pin the intended language with the agent's " + + "explicit response-locale setting if the inferred target is wrong; a persistent mismatch here costs " + + "an extra model call and breaks the prompt cache each turn." + : "The enforced locale is an OPERATOR PIN (localeSource=explicit) and the model did not honour it. " + + "Verify the pin is the language you intend, then inspect the selected model's locale fidelity."; +} diff --git a/packages/agent/src/executor/resolve-response-locale-policy.test.ts b/packages/agent/src/executor/resolve-response-locale-policy.test.ts index 9e6aa4d076..49a9b973a3 100644 --- a/packages/agent/src/executor/resolve-response-locale-policy.test.ts +++ b/packages/agent/src/executor/resolve-response-locale-policy.test.ts @@ -34,11 +34,15 @@ describe("resolveResponseLocalePolicy", () => { }); }); - it("derives an open undetermined-language script tag from the current request text", () => { + // The script-derived tier still INFERS the locale (it rides the prompt as a hint), + // but it no longer ENFORCES it: inferring from the current message alone let a + // single message switch a whole conversation's language and then burn a repair + // round-trip fighting the model. Only an operator pin enforces. + it("derives an open undetermined-language script tag from the current request text, ADVISORY only", () => { expect(resolveResponseLocalePolicy({ requestText: "اكتب ملخصًا قصيرًا" })).toEqual({ locale: "und-Arab", source: "request", - enforceLocale: true, + enforceLocale: false, }); expect(resolveResponseLocalePolicy({ requestText: "10978704" })).toEqual({ source: "unset", @@ -150,7 +154,8 @@ describe("request-locale vs conversation-script precedence", () => { }); expect(policy.locale).toBe("und-Hebr"); expect(policy.source).toBe("request"); - expect(policy.enforceLocale).toBe(true); + // Advisory — the script-derived tier informs but never enforces. + expect(policy.enforceLocale).toBe(false); }); it("keeps the request locale when it agrees with the message script", () => { @@ -163,14 +168,14 @@ describe("request-locale vs conversation-script precedence", () => { expect(policy.enforceLocale).toBe(true); }); - it("enforces the current Latin prose script when it contradicts a non-Latin transport locale", () => { + it("prefers the current Latin prose script over a contradicting non-Latin transport locale (advisory)", () => { const policy = resolveResponseLocalePolicy({ requestLocale: "he", requestText: "What is the weather tomorrow in Tel Aviv?", }); expect(policy.locale).toBe("und-Latn"); expect(policy.source).toBe("request"); - expect(policy.enforceLocale).toBe(true); + expect(policy.enforceLocale).toBe(false); }); it("does not treat a short Latin identifier as a conversation-language override", () => { @@ -203,3 +208,34 @@ describe("request-locale vs conversation-script precedence", () => { expect(policy.enforceLocale).toBe(true); }); }); + +describe("an operator pin is the ONLY enforcer of response locale", () => { + // Decision (owner, 2026-07-26): pin the operator's language; remove per-message + // locale switching. Live, one English instruction inside an otherwise-Hebrew + // conversation set `locale=en enforce=true`, and three repair passes each cost a + // model call plus a prompt-cache break ($1.72) while the model correctly kept + // answering in Hebrew. + it("an English message in a Hebrew conversation does NOT enforce English", () => { + const policy = resolveResponseLocalePolicy({ + requestText: "Install this skill: https://example.invalid/skills/xlsx", + }); + expect(policy.enforceLocale).toBe(false); + }); + + it("an explicit operator pin DOES enforce, and outranks the message script", () => { + const policy = resolveResponseLocalePolicy({ + explicitLocale: "he-IL", + requestText: "Install this skill: https://example.invalid/skills/xlsx", + }); + expect(policy.locale).toBe("he-IL"); + expect(policy.source).toBe("explicit"); + expect(policy.enforceLocale).toBe(true); + }); + + it("no pin and no script signal stays unset (nothing to enforce)", () => { + expect(resolveResponseLocalePolicy({ requestText: "10978704" })).toEqual({ + source: "unset", + enforceLocale: false, + }); + }); +}); diff --git a/packages/agent/src/executor/resolve-response-locale-policy.ts b/packages/agent/src/executor/resolve-response-locale-policy.ts index 4c7cc17c6c..6769b50991 100644 --- a/packages/agent/src/executor/resolve-response-locale-policy.ts +++ b/packages/agent/src/executor/resolve-response-locale-policy.ts @@ -72,7 +72,20 @@ export function resolveResponseLocalePolicy( locale: requestScriptLocale, source: "request", ...(translationTarget === undefined ? {} : { translationTarget }), - enforceLocale: true, + // ADVISORY, never ENFORCED. This tier infers a locale from the script of the + // CURRENT message alone, so enforcing it made a single message switch the + // whole conversation's language — and then spend a repair round-trip trying + // to make the model comply. Live: one English instruction inside an + // otherwise-Hebrew conversation set `locale=en enforce=true`; all three + // repair passes correctly came back Hebrew (the model was honouring the + // established language), each costing a full extra model call and a + // prompt-cache break. + // + // Only an OPERATOR PIN (`explicitLocale` → source "explicit") enforces. The + // inferred locale still rides the prompt as a hint, which is all it was ever + // reliable enough to be: a model already mirrors its interlocutor's language + // without being policed into it. + enforceLocale: false, }; } return { diff --git a/packages/agent/src/executor/stream-wrappers/request-body/prefix-stability.test.ts b/packages/agent/src/executor/stream-wrappers/request-body/prefix-stability.test.ts index d767695065..fe9832d26c 100644 --- a/packages/agent/src/executor/stream-wrappers/request-body/prefix-stability.test.ts +++ b/packages/agent/src/executor/stream-wrappers/request-body/prefix-stability.test.ts @@ -27,6 +27,33 @@ function noopLogger() { } describe("classifyPrefixMutation — diagnostic honesty", () => { + + // LIVE INCIDENT (comis-moshe 2026-07-26): 30 "Unstable prefix detected" WARNs, and + // 29 of the 31 counted `cache_prefix_churn` signals were labelled "unknown" — while + // the signature it printed already carried the answer: + // assistant|b1|t0|r0|len590 -> assistant|b2|t0|r0|len590 + // assistant|b1|t0|r0|len240 -> assistant|b2|t0|r0|len30 + // The BLOCK COUNT moved. `parseSig` never parsed `b`, so a pure block-count + // reshape (same role, thinking unchanged, <500 char delta) fell through to + // "unknown" — leaving the dominant churn cause unnamed on every occurrence. + it("names a block-count reshape instead of returning 'unknown' (same role, same length)", () => { + const msg = { role: "assistant", content: [{ type: "text", text: "x" }] }; + const cls = classifyPrefixMutation(msg, "assistant|b1|t0|r0|len590", "assistant|b2|t0|r0|len590"); + expect(cls).not.toBe("unknown"); + expect(cls).toContain("block-count-changed"); + }); + + it("names a block-count reshape when the length ALSO moved but under the content-cleared threshold", () => { + const msg = { role: "assistant", content: [{ type: "text", text: "x" }] }; + const cls = classifyPrefixMutation(msg, "assistant|b1|t0|r0|len240", "assistant|b2|t0|r0|len30"); + expect(cls).toContain("block-count-changed"); + }); + + it("does NOT invent the class when the block count is stable", () => { + const msg = { role: "assistant", content: [{ type: "text", text: "x" }] }; + const cls = classifyPrefixMutation(msg, "assistant|b2|t0|r0|len100", "assistant|b2|t0|r0|len100"); + expect(cls).not.toContain("block-count-changed"); + }); it("labels a ROLE CHANGE (assistant→user) as a structural index-shift, not datetime-preamble", () => { // The exact comis-harel sig: an empty assistant tool-use turn at an index // becomes a user turn carrying the dynamic preamble (which contains the diff --git a/packages/agent/src/executor/stream-wrappers/request-body/prefix-stability.ts b/packages/agent/src/executor/stream-wrappers/request-body/prefix-stability.ts index 2282edc335..5170ab2b0c 100644 --- a/packages/agent/src/executor/stream-wrappers/request-body/prefix-stability.ts +++ b/packages/agent/src/executor/stream-wrappers/request-body/prefix-stability.ts @@ -75,10 +75,22 @@ function structEachMessage(messages: Array>, endIdx: num } /** Parse the `t`/`r`/`len` fields out of a struct sig (returns {t,r,len} or undefined). */ -function parseSig(sig: string | undefined): { t: number; r: number; len: number } | undefined { +function parseSig( + sig: string | undefined, +): { b: number; t: number; r: number; len: number } | undefined { if (!sig) return undefined; + // `b` (content BLOCK count) was previously not parsed at all, which made the + // classifier structurally blind to a block-count reshape — the dominant live + // churn cause, reported as "unknown" on 29 of 31 signals while the printed + // signature already showed b1→b2 (comis-moshe 2026-07-26). + const b = /\|b(\d+)\|/.exec(sig); const t = /\|t(\d+)\|/.exec(sig); const r = /\|r(\d+)\|/.exec(sig); const len = /\|len(\d+)$/.exec(sig); - return { t: t ? Number(t[1]) : 0, r: r ? Number(r[1]) : 0, len: len ? Number(len[1]) : 0 }; + return { + b: b ? Number(b[1]) : 0, + t: t ? Number(t[1]) : 0, + r: r ? Number(r[1]) : 0, + len: len ? Number(len[1]) : 0, + }; } /** @@ -132,6 +144,13 @@ export function classifyPrefixMutation( if (!roleChanged && p.r > c.r) return "inline-recall"; if (p.t > c.t) classes.push("thinking-cleared"); else if (p.len - c.len > 500) classes.push("content-cleared"); + // A CACHED assistant message whose content-block COUNT moved between turns was + // re-serialized with a different block shape (text split/merged, a replay block + // added or stripped). Same role, thinking unchanged, length delta below the + // content-cleared threshold — so none of the classes above catch it, and it + // used to render as "unknown". It is a real, actionable cause: the cached + // prefix must be byte-stable, so a reshape re-pays the whole prefix. + if (p.b !== c.b) classes.push("block-count-changed"); } // Content-pattern classes (per-request-varying injected content). diff --git a/packages/agent/src/model/fresh-tail-clamp.test.ts b/packages/agent/src/model/fresh-tail-clamp.test.ts index dd2a01ab15..abe5f9173e 100644 --- a/packages/agent/src/model/fresh-tail-clamp.test.ts +++ b/packages/agent/src/model/fresh-tail-clamp.test.ts @@ -72,4 +72,77 @@ describe("resolveClampedFreshTailTurns", () => { expect(result).toBeGreaterThanOrEqual(1); expect(result).toBeLessThanOrEqual(30); }); + + // --------------------------------------------------------------------------- + // THE PRODUCTION PATH (comis-moshe 2026-07-26). + // + // Every test above pins the clamp with an EXPLICIT avgTokensPerStep — but + // `lcd-assembler.ts` calls it with TWO arguments, so production always takes + // the auto-estimate branch. That branch estimated a step at `W/20` — 5% of the + // window — which SCALES WITH W, so the ratio 0.3W / (W/20) cancels and the + // result was the constant 6 for EVERY finite window and EVERY configured + // value. `contextEngine.freshTailTurns` (schema 1..50, default 8) was + // therefore unreachable above 6 on every deployment. + // + // Live consequence: on a 1M-token window with 87,740 tokens in use (9%), a turn + // with 4 background-tool cycles (8 steps) slid the user's ORIGINATING request out + // of the verbatim tail — and the agent apologized to the user for work they had + // explicitly requested. The two smoke tests above could not catch it: both + // assertions (>=1, <=configured) hold for a constant 6. + // --------------------------------------------------------------------------- + + it("does NOT clamp the DEFAULT 8 on a frontier 1M window (the live shape)", () => { + expect(resolveClampedFreshTailTurns(1_000_000, 8)).toBe(8); + }); + + it("honors a RAISED freshTailTurns on a large window", () => { + expect(resolveClampedFreshTailTurns(1_000_000, 20)).toBe(20); + expect(resolveClampedFreshTailTurns(1_000_000, 50)).toBe(50); + }); + + it("does not clamp the default on the common 128K/200K windows", () => { + expect(resolveClampedFreshTailTurns(128_000, 8)).toBe(8); + expect(resolveClampedFreshTailTurns(200_000, 8)).toBe(8); + }); + + it("is NOT window-independent — a bigger window must afford at least as many steps", () => { + // The bug's signature: the auto-estimate returned the SAME number for every + // window. Monotonic non-decreasing in W is the invariant that kills it. + const windows = [8_000, 32_000, 128_000, 200_000, 1_000_000, 2_000_000]; + const results = windows.map((w) => resolveClampedFreshTailTurns(w, 50)); + for (let i = 1; i < results.length; i += 1) { + expect(results[i]!).toBeGreaterThanOrEqual(results[i - 1]!); + } + // …and it must actually VARY (a constant is monotonic too). + expect(new Set(results).size).toBeGreaterThan(1); + }); + + it("STILL clamps on a genuinely small window (the clamp keeps its purpose)", () => { + // A tiny window cannot let the verbatim tail eat it. + expect(resolveClampedFreshTailTurns(4_096, 8)).toBeLessThan(8); + expect(resolveClampedFreshTailTurns(4_096, 8)).toBeGreaterThanOrEqual(1); + // …and it floors at 1, never 0. + expect(resolveClampedFreshTailTurns(1_000, 8)).toBe(1); + }); + + it("is NON-REGRESSING at the smallest viable window: 8192 still yields the previous 6", () => { + // The calibration constant is chosen so no existing deployment loses tail + // steps — floor(0.3 * 8192 / 400) = 6, exactly what the buggy constant gave. + expect(resolveClampedFreshTailTurns(8_192, 8)).toBe(6); + }); + + it("never returns FEWER steps than the pre-fix constant 6 at any viable window", () => { + for (const w of [8_192, 16_000, 32_000, 128_000, 200_000, 1_000_000]) { + expect(resolveClampedFreshTailTurns(w, 8)).toBeGreaterThanOrEqual(Math.min(8, 6)); + } + }); + + it("never exceeds the configured value, on any window", () => { + for (const w of [1_000, 8_000, 32_000, 128_000, 1_000_000]) { + for (const c of [1, 4, 8, 20, 50]) { + expect(resolveClampedFreshTailTurns(w, c)).toBeLessThanOrEqual(c); + expect(resolveClampedFreshTailTurns(w, c)).toBeGreaterThanOrEqual(1); + } + } + }); }); diff --git a/packages/agent/src/model/fresh-tail-clamp.ts b/packages/agent/src/model/fresh-tail-clamp.ts index 38099ebc81..0ec0b3945a 100644 --- a/packages/agent/src/model/fresh-tail-clamp.ts +++ b/packages/agent/src/model/fresh-tail-clamp.ts @@ -18,6 +18,55 @@ * @module */ +/** + * The fraction of the effective window the verbatim fresh tail may occupy on its + * own. Conservative placeholder pending real recall-impact measurement — do not + * treat it as a calibrated constant. + */ +const FRESH_TAIL_WINDOW_FRACTION = 0.3; + +/** + * Absolute per-step token estimate used when the caller supplies none. + * + * ⚠ This MUST NOT be derived from `effectiveWindow`. The original code estimated + * a step at `floor(effectiveWindow / 20)` — 5% of the window — which cancels + * against the 30% budget: + * + * affordable = floor(0.3·W / (W/20)) = floor(0.3 · 20) = 6 ∀ finite W + * + * so the clamp returned the constant **6** for every window and every configured + * value, making `contextEngine.freshTailTurns` (schema 1..50, default 8) + * unreachable above 6 on every deployment. Live consequence (comis-moshe + * 2026-07-26): on a 1M-token window with 9% of it in use, a turn with four + * background-tool cycles slid the user's own request out of the verbatim tail and + * the agent apologized for work the user had explicitly asked for. The two + * existing auto-estimate tests could not catch it — both only asserted + * `1 <= result <= configured`, which a constant 6 satisfies. + * + * CALIBRATION — chosen so this change can only ever keep MORE context, never + * less. At the smallest window Comis treats as viable (8,192) the value + * reproduces the previous behaviour exactly: + * + * floor(0.3 · 8192 / 400) = floor(6.14) = 6 ← the old constant + * + * so no deployment loses tail steps, and every window above ~11K now reaches its + * configured value instead of being pinned at 6. Measured live step sizes on the + * incident turn were `[1385, 333, 4018, 193, 49, 178, 49]` tokens (mean ≈ 890), + * so 400 deliberately UNDER-estimates a step: this is the coarse guard, and + * under-estimating errs toward keeping the user's own message in context. The + * PRECISE enforcement — which is what actually protects a small window where the + * system prompt dominates — is `boundFreshTailTotalToResidual` in the assembler + * (a total-token bound applied after the per-message char cap), not this clamp. + * + * Resulting behaviour (default `configuredTurns` = 8): + * W = 8,192 → 6 (identical to the pre-fix behaviour) + * W = 16,000 → 8 + * W = 32,000 → 8 + * W = 128,000 → 8 + * W = 1,000,000 → 8 (and 50 configured → 50) + */ +const COARSE_TOKENS_PER_STEP_ESTIMATE = 400; + /** * Resolve a clamped fresh-tail turn count from the effective context window. * @@ -26,7 +75,9 @@ * @param configuredTurns - The operator-configured freshTailTurns value * (from schema default 8, min 1, max 50). * @param avgTokensPerStep - Optional estimated tokens per conversation step. - * If omitted, estimated as `max(1, floor(effectiveWindow / 20))` (5% floor). + * If omitted, {@link COARSE_TOKENS_PER_STEP_ESTIMATE} is used — an ABSOLUTE + * token estimate, deliberately NOT a fraction of the window (see that + * constant for why a window-relative estimate made this clamp a constant). * @returns The clamped turn count — always ≥ 1, always ≤ configuredTurns. * * NB: this turn-count clamp is the COARSE upper bound only. @@ -43,12 +94,13 @@ export function resolveClampedFreshTailTurns( avgTokensPerStep?: number, ): number { if (!isFinite(effectiveWindow)) return configuredTurns; // frontier/mid: byte-identical - // Estimate tokens per step if not provided (5% of effectiveWindow as single-step floor) + // The per-step estimate MUST be absolute. A window-relative estimate makes the + // whole clamp a constant (see COARSE_TOKENS_PER_STEP_ESTIMATE). const tokPerStep = avgTokensPerStep !== undefined && avgTokensPerStep > 0 ? avgTokensPerStep - : Math.max(1, Math.floor(effectiveWindow / 20)); - const budget = Math.floor(effectiveWindow * 0.3); + : COARSE_TOKENS_PER_STEP_ESTIMATE; + const budget = Math.floor(effectiveWindow * FRESH_TAIL_WINDOW_FRACTION); const affordable = Math.max(1, Math.floor(budget / tokPerStep)); return Math.min(configuredTurns, affordable); } diff --git a/packages/cli/src/commands/mcp-token.test.ts b/packages/cli/src/commands/mcp-token.test.ts new file mode 100644 index 0000000000..687f8c98a9 --- /dev/null +++ b/packages/cli/src/commands/mcp-token.test.ts @@ -0,0 +1,71 @@ +// SPDX-License-Identifier: Apache-2.0 +/** + * `ensureGatewayToken` — the pre-client bearer-token resolution and, critically, + * what it tells an operator when it cannot resolve one. + */ +import { describe, it, expect, beforeEach, afterEach } from "vitest"; +import { ensureGatewayToken } from "./mcp-token.js"; + +const KEY = "COMIS_GATEWAY_TOKEN"; + +describe("ensureGatewayToken", () => { + let saved: string | undefined; + beforeEach(() => { + saved = process.env[KEY]; + delete process.env[KEY]; + }); + afterEach(() => { + if (saved === undefined) delete process.env[KEY]; + else process.env[KEY] = saved; + }); + + it("threads an explicit --token through to the environment", () => { + ensureGatewayToken("literal-token-value-at-least-32-chars-x"); + expect(process.env[KEY]).toBe("literal-token-value-at-least-32-chars-x"); + }); + + it("accepts an already-present environment value", () => { + process.env[KEY] = "already-set-token-value"; + expect(() => ensureGatewayToken(undefined)).not.toThrow(); + }); + + describe("the miss message", () => { + // Live friction: on a healthy box running `security.storage: encrypted`, + // `comis mcp list` failed with "Missing COMIS_GATEWAY_TOKEN — set in + // ~/.comis/.env … Hint: run `comis init` to generate a gateway token" — while + // the token was present in secrets.db and the daemon was serving with it. + // Following that hint would have ROTATED the live token. + function missMessage(): string { + // Guarantee a miss regardless of the developer's own ~/.comis/.env by + // asserting only on the thrown text when the env var is absent. + try { + ensureGatewayToken(undefined); + return ""; + } catch (e) { + return e instanceof Error ? e.message : String(e); + } + } + + it("does NOT lead with `comis init` as the fix", () => { + const msg = missMessage(); + if (msg === "") return; // developer box has a token in ~/.comis/.env + const initIdx = msg.indexOf("comis init"); + const secretsIdx = msg.indexOf("comis secrets get"); + expect(secretsIdx, "the read-only recovery must be present").toBeGreaterThanOrEqual(0); + expect(initIdx, "`comis init` must come AFTER the safe recovery").toBeGreaterThan(secretsIdx); + }); + + it("warns that `comis init` is destructive on a serving box", () => { + const msg = missMessage(); + if (msg === "") return; + expect(msg).toMatch(/NEW token|break a daemon/i); + }); + + it("names the encrypted store as the place the token actually lives", () => { + const msg = missMessage(); + if (msg === "") return; + expect(msg).toContain("secrets.db"); + expect(msg).toContain("security.storage: encrypted"); + }); + }); +}); diff --git a/packages/cli/src/commands/mcp-token.ts b/packages/cli/src/commands/mcp-token.ts index 85cd957670..66d39dc30f 100644 --- a/packages/cli/src/commands/mcp-token.ts +++ b/packages/cli/src/commands/mcp-token.ts @@ -55,8 +55,19 @@ export function ensureGatewayToken(flagToken: string | undefined): void { loadEnvFile(join(homedir(), ".comis", ".env")); const existing = systemGetEnv("COMIS_GATEWAY_TOKEN"); if (existing !== undefined && existing.length > 0) return; + // The old hint said "run `comis init` to generate a gateway token". Under + // `security.storage: encrypted` — the recommended posture — the token is NOT in + // `~/.comis/.env` (that file holds only SECRETS_MASTER_KEY); it lives in the + // encrypted `secrets.db`, which is why the daemon boots fine while this + // resolver misses. Following that hint on a healthy box ROTATES the live + // token and breaks the running daemon. Lead with the read-only recovery. throw new Error( - "Missing COMIS_GATEWAY_TOKEN — set in ~/.comis/.env or pass --token .\n" + - "Hint: run `comis init` to generate a gateway token, or `comis pm2 setup` to bootstrap the environment file.", + "Missing COMIS_GATEWAY_TOKEN in the environment.\n" + + "If the daemon is running, a token already exists — it is just not in the environment:\n" + + " • encrypted storage (`security.storage: encrypted`): the token is in `secrets.db`, not `~/.comis/.env`.\n" + + " Read it and pass it through: comis --token \"$(comis secrets get COMIS_GATEWAY_TOKEN)\" \n" + + " • file storage: export COMIS_GATEWAY_TOKEN, or add it to `~/.comis/.env`.\n" + + "Only run `comis init` on a box with NO working install — it generates a NEW token and " + + "will break a daemon that is already serving with the current one.", ); } diff --git a/packages/cli/src/commands/system-health.ts b/packages/cli/src/commands/system-health.ts index e877d181a1..1699d0a0da 100644 --- a/packages/cli/src/commands/system-health.ts +++ b/packages/cli/src/commands/system-health.ts @@ -94,9 +94,14 @@ export function registerSystemHealthCommand(program: Command): void { // reply) vs delivered-with-tool-errors (a final answer WAS delivered // despite a recovered/acknowledged tool error) so a system of self-healed // hiccups is not misread as a high failure rate. + // Read the split OFF the report — deriving it here is what let the table + // and the report's own JSON disagree (comis-moshe 2026-07-26). The `??` + // fallbacks keep an older daemon's report renderable. const delivered = report.sessions.deliveredWithToolErrors ?? 0; - const hardDegraded = report.sessions.degraded - delivered; - const hardPct = report.sessions.total > 0 ? Math.round((hardDegraded / report.sessions.total) * 100) : 0; + const hardDegraded = report.sessions.hardDegraded ?? (report.sessions.degraded - delivered); + const hardRate = report.sessions.hardDegradedRate + ?? (report.sessions.total > 0 ? hardDegraded / report.sessions.total : 0); + const hardPct = Math.round(hardRate * 100); info( delivered > 0 ? `Sessions: ${report.sessions.total} (${hardDegraded} hard-degraded, ${hardPct}%; +${delivered} delivered-with-tool-errors — user still got a reply)` diff --git a/packages/core/src/activity/index.ts b/packages/core/src/activity/index.ts index 37d5e9cf0c..e81a59b1bb 100644 --- a/packages/core/src/activity/index.ts +++ b/packages/core/src/activity/index.ts @@ -27,7 +27,7 @@ export { ApprovalChoiceSchema, ApprovalCorrelationSchema } from "./approval.js"; export type { ApprovalCorrelation, ApprovalChoice } from "./approval.js"; // --- turn-outcome -------------------------------------------------------------- -export { isNonEmptyEvents } from "./turn-outcome.js"; +export { isNonEmptyEvents, UNATTRIBUTED_FAILURE_REASON } from "./turn-outcome.js"; export type { FinalDeliveryReceipt, DeliveryFailureReceipt, diff --git a/packages/core/src/activity/turn-outcome.ts b/packages/core/src/activity/turn-outcome.ts index 3e44dfade4..7e2a7df865 100644 --- a/packages/core/src/activity/turn-outcome.ts +++ b/packages/core/src/activity/turn-outcome.ts @@ -53,6 +53,20 @@ export interface DeliveryFailureReceipt { */ export type DeliveryStageResult = Result; +/** + * The fixed `reason` for a `failure` outcome that carries NO failed activity + * event — the turn failed, but nothing on the user-visible tool timeline can be + * named as the cause. + * + * Without it, `failureLabel()` renders the bare `" {errorKind}"`, which + * reached a real user as a chat bubble whose entire text was "❌ dependency" + * (comis-moshe 2026-07-26). That happened because every tool that actually failed + * had been closed `status:"completed"` at background hand-off, leaving + * `failedEvents: []` on a genuine failure. A closed-vocabulary named constant — + * never raw provider/internal text. + */ +export const UNATTRIBUTED_FAILURE_REASON = "a step failed outside the tool timeline"; + export type TurnOutcome = | { kind: "success"; trivial: boolean; delivery: FinalDeliveryReceipt } | { kind: "success_with_recovered_failures"; trivial: false; diff --git a/packages/core/src/api-contracts/incident-report-sections.ts b/packages/core/src/api-contracts/incident-report-sections.ts index 3c48fc196f..0268962af6 100644 --- a/packages/core/src/api-contracts/incident-report-sections.ts +++ b/packages/core/src/api-contracts/incident-report-sections.ts @@ -51,6 +51,19 @@ export const IncidentContextBudgetSchema = z.object({ outputHeadroom: z.number(), /** Fit-check outcome. */ verdict: z.enum(["fits", "downshifted", "exhausted"]), + /** + * Trailing conversation STEPS kept verbatim this turn, EFFECTIVE (post-clamp). + * + * The verbatim tail is bounded by step count, not tokens: a turn with more + * tool round-trips than this slides the user's ORIGINATING request out of + * verbatim context — while `verdict` still reads "fits". Live, that produced + * an agent that apologized for work the user had explicitly requested, with + * 91% of the window unused (comis-moshe 2026-07-26). Optional (additive). + */ + freshTailSteps: z.number().optional(), + /** The configured `contextEngine.freshTailTurns`. A value BELOW this means the + * operator's knob was clamped — see {@link freshTailSteps}. */ + freshTailStepsConfigured: z.number().optional(), }); /** The per-call context budget equation (see {@link IncidentContextBudgetSchema}). */ diff --git a/packages/core/src/api-contracts/incident-report.ts b/packages/core/src/api-contracts/incident-report.ts index 0be18f8c69..11ebe62e32 100644 --- a/packages/core/src/api-contracts/incident-report.ts +++ b/packages/core/src/api-contracts/incident-report.ts @@ -76,7 +76,23 @@ export const IncidentReportSchema = z.object({ }), cost: z.object({ costUsd: z.number(), + /** + * ALL tokens the session moved, INCLUDING cache reads and cache writes. + * + * ⚠ This is NOT the same quantity as `SystemHealthReport.cost.totalTokens`, + * which counts input+output only and EXCLUDES cache. One 27-minute session + * legitimately reported 6,043,245 here and 18,637 there, and nothing in + * either JSON said so — a reader comparing the two lenses concludes one is + * broken. The `tokenBasis` discriminator below states which convention this + * number follows so the lenses reconcile without reading the source. + */ totalTokens: z.number(), + /** + * The counting convention `totalTokens` follows. Closed union so a consumer + * can reconcile lenses programmatically rather than by convention. + * Optional (additive; an older producer omits it). + */ + tokenBasis: z.literal("input+output+cache").optional(), cacheReadRatio: z.number(), }), timing: z.object({ diff --git a/packages/core/src/api-contracts/system-health-report.ts b/packages/core/src/api-contracts/system-health-report.ts index 87eccdc5bc..1271d9af8f 100644 --- a/packages/core/src/api-contracts/system-health-report.ts +++ b/packages/core/src/api-contracts/system-health-report.ts @@ -48,6 +48,20 @@ export const SystemHealthReportSchema = z.object({ * a JSON consumer / the CLI. Optional (additive; pre-existing readers ignore it). */ deliveredWithToolErrors: z.number().optional(), + /** + * `degraded − deliveredWithToolErrors` — the sessions where the user did NOT + * get a good reply. This is the count the degradation FINDINGS fire on, so it + * is the number that decides whether the system is healthy. + * + * It lives in the report (rather than being re-derived per renderer) because + * it was not: the CLI computed it locally and printed "0 hard-degraded, 0%" + * while the JSON of the SAME report carried `degraded:1, degradedRate:1` + * (comis-moshe 2026-07-26). Two renderings of one report must not disagree. + * Optional (additive; pre-existing readers ignore it). + */ + hardDegraded: z.number().optional(), + /** `hardDegraded / total` (0 when `total` is 0). See {@link hardDegraded}. */ + hardDegradedRate: z.number().optional(), }), /** Merged across the window + capped top-N (counts only — no raw bodies). */ topErrorKinds: z.array(z.object({ kind: z.string(), count: z.number() })), @@ -65,7 +79,22 @@ export const SystemHealthReportSchema = z.object({ toolStats: z.record(z.string(), z.object({ ok: z.number(), failed: z.number() })), cost: z.object({ costUsd: z.number(), + /** + * Billable tokens in the window: input + output ONLY — cache reads and cache + * writes are EXCLUDED. + * + * ⚠ This is NOT the same quantity as `IncidentReport.cost.totalTokens`, which + * INCLUDES cache. One 27-minute session legitimately reported 18,637 here and + * 6,043,245 there; with both fields named `totalTokens` and neither stating + * its convention, a reader comparing the two lenses concludes one is broken. + * The `tokenBasis` discriminator states which convention this number follows. + */ totalTokens: z.number(), + /** + * The counting convention `totalTokens` follows. Closed union so a consumer + * can reconcile lenses programmatically. Optional (additive). + */ + tokenBasis: z.literal("input+output").optional(), /** * Off-session (background-job) LLM spend in the window — reflection cron * runs et al. that key their token usage to a synthetic `__PREFIX__` diff --git a/packages/core/src/config/index.ts b/packages/core/src/config/index.ts index 715cdd4b75..f733f36b1c 100644 --- a/packages/core/src/config/index.ts +++ b/packages/core/src/config/index.ts @@ -180,6 +180,7 @@ export { MediaPersistenceConfigSchema, ImageGenerationConfigSchema, VideoGenerationConfigSchema, + MCP_CALL_TOOL_TIMEOUT_MS_DEFAULT, } from "./schema-integrations.js"; export { MonitoringConfigSchema } from "./schema-observability.js"; export { ObservabilityConfigSchema, SpendConfigSchema } from "./schema-observability.js"; diff --git a/packages/core/src/config/schema-integrations.ts b/packages/core/src/config/schema-integrations.ts index 32f1124ffd..81de09bea9 100644 --- a/packages/core/src/config/schema-integrations.ts +++ b/packages/core/src/config/schema-integrations.ts @@ -197,6 +197,17 @@ export const McpServerEntrySchema = z.preprocess( }), ); +/** + * Default MCP tool-call deadline, in milliseconds. + * + * Named + exported so the client manager's construction-time fallback cannot + * drift from the schema (it did: the client fell back to 60_000 while the + * schema defaulted to 120_000, so an un-threaded wiring silently halved every + * deployment's deadline). `test/architecture/mcp-timeout-default-parity.test.ts` + * pins the two together. + */ +export const MCP_CALL_TOOL_TIMEOUT_MS_DEFAULT = 120_000; + /** * MCP integration configuration. */ @@ -205,7 +216,7 @@ export const McpConfigSchema = z.strictObject({ servers: z.array(McpServerEntrySchema).default([]), /** Default timeout for MCP tool calls in milliseconds (default: 120000). * Image generation and other slow tools may need 2+ minutes. */ - callToolTimeoutMs: z.number().int().positive().default(120_000), + callToolTimeoutMs: z.number().int().positive().default(MCP_CALL_TOOL_TIMEOUT_MS_DEFAULT), /** Default max concurrent tool calls for stdio servers (default: 1). */ stdioDefaultConcurrency: z.number().int().positive().default(1), /** Default max concurrent tool calls for HTTP/SSE servers (default: 4). */ diff --git a/packages/core/src/event-bus/events-infra.ts b/packages/core/src/event-bus/events-infra.ts index 015c97ed0b..f125ee6bb5 100644 --- a/packages/core/src/event-bus/events-infra.ts +++ b/packages/core/src/event-bus/events-infra.ts @@ -655,6 +655,8 @@ export interface InfraEvents { durationMs: number; origin: BackgroundTaskOrigin; timestamp: number; + /** {@link BACKGROUND_TASK_DISPATCH_REDELIVERY} — see the failed variant. */ + dispatchRedelivery?: boolean; }; /** Background task failed (timeout, error, or daemon restart). @@ -668,6 +670,18 @@ export interface InfraEvents { durationMs: number; origin: BackgroundTaskOrigin; timestamp: number; + /** + * TRUE when this emission is a `scheduleDispatchRetry` backoff tick + * re-driving DISPATCH for an already-terminal task — not a new failure. + * + * The retry re-emits the terminal event because that is how the dispatch + * listeners are woken; without this marker the trajectory recorded one line + * per tick, so ONE failure read as many (live: 55 records for 4 tasks, + * 17/15/13/10). Consumers that count occurrences — the trajectory bridge + * above all — must skip a redelivery. Same posture as + * `background_task:notified.trajectoryRecorded`. + */ + dispatchRedelivery?: boolean; }; /** diff --git a/packages/core/src/event-bus/events-messaging.ts b/packages/core/src/event-bus/events-messaging.ts index dc387a6b1b..01a3074b0b 100644 --- a/packages/core/src/event-bus/events-messaging.ts +++ b/packages/core/src/event-bus/events-messaging.ts @@ -442,6 +442,24 @@ export interface MessagingEvents { outputHeadroom: number; /** Fit-check outcome (closed union — never an open string). */ verdict: "fits" | "downshifted" | "exhausted"; + /** + * The EFFECTIVE number of trailing conversation STEPS kept verbatim this + * turn (post-clamp), and the operator-CONFIGURED value it was clamped from. + * + * The fresh tail is bounded by STEP COUNT, not tokens — so on a long tool + * loop the user's originating request slides out of verbatim context while + * `verdict` still reads "fits" and the window sits nearly empty. That is + * exactly what happened live (comis-moshe 2026-07-26): the agent apologized + * for work the user HAD requested, with 87,740 of 1,000,000 tokens in use + * and `droppedCount:0`. Without these two numbers the report shows a clean + * "fits" and the slide is invisible. + * + * Optional (additive) — emitters that cannot resolve the step bound omit + * both rather than reporting a wrong number. + */ + freshTailSteps?: number; + /** The configured `contextEngine.freshTailTurns`. See {@link freshTailSteps}. */ + freshTailStepsConfigured?: number; }; /** A non-Latin search returned zero hits on a CLEANLY-executed lane. diff --git a/packages/core/src/exports/activity.ts b/packages/core/src/exports/activity.ts index 1dc98cafd8..79c80ccc18 100644 --- a/packages/core/src/exports/activity.ts +++ b/packages/core/src/exports/activity.ts @@ -17,6 +17,7 @@ export { ApprovalCorrelationSchema, // turn-outcome isNonEmptyEvents, + UNATTRIBUTED_FAILURE_REASON, // template-engine / semantic-classifier / label-spec applyTemplate, classifySemanticPhase, diff --git a/packages/core/src/exports/config.ts b/packages/core/src/exports/config.ts index 868547a013..97ae9d1d94 100644 --- a/packages/core/src/exports/config.ts +++ b/packages/core/src/exports/config.ts @@ -48,6 +48,7 @@ export { BraveSearchConfigSchema, McpServerEntrySchema, McpConfigSchema, + MCP_CALL_TOOL_TIMEOUT_MS_DEFAULT, TranscriptionConfigSchema, TtsConfigSchema, TtsAutoModeSchema, diff --git a/packages/core/src/security/secret-detection.test.ts b/packages/core/src/security/secret-detection.test.ts index 5987e71614..e2548c2081 100644 --- a/packages/core/src/security/secret-detection.test.ts +++ b/packages/core/src/security/secret-detection.test.ts @@ -96,6 +96,110 @@ describe("isSecretFieldName — superset", () => { } }); + // --------------------------------------------------------------------------- + // The `.*token` WILDCARD over-match. + // + // `range_token` — an MCP tool argument holding a plain enum value — matched + // `/^.*token$/i`, so LCD/session persistence rewrote it to "[REDACTED]". The + // model then read the placeholder back out of its own replay context and sent + // the literal string "[REDACTED]" to the server, which rejected it + // (`invalid_enum_value received:"[REDACTED]"`). The agent reported that its own + // argument was "exposed to me redacted" and abandoned the cheap query path for a + // far more expensive one that then timed out repeatedly. + // + // The fix is an EXACT-NAME exception set, never a loosening of the pattern — + // every entry below is a non-credential whose name merely ends in "token(s)". + // The negative matrix that follows is the load-bearing half. + // --------------------------------------------------------------------------- + + it("does NOT flag non-credential names the .*token wildcard over-matches", () => { + for (const name of [ + "range_token", // the live case: an enum selector ("last_week") + "rangeToken", + "max_tokens", // provider request knob + "maxTokens", + "num_tokens", + "tokens", + "token_count", + "tokenCount", + "input_tokens", + "output_tokens", + "total_tokens", + "cache_read_input_tokens", + "prompt_tokens", + "completion_tokens", + "token_limit", + "tokenizer", + ]) { + expect(isSecretFieldName(name), `${name} must NOT be treated as a secret`).toBe(false); + } + }); + + it("STILL flags every credential-bearing token name (the exception set must not leak)", () => { + for (const name of [ + "token", + "botToken", + "bot_token", + "accessToken", + "access_token", + "refresh_token", + "refreshToken", + "id_token", + "auth_token", + "authToken", + "api_token", + "apiToken", + "bearer_token", + "session_token", + "gateway_token", + "COMIS_GATEWAY_TOKEN", + "TELEGRAM_BOT_TOKEN", + "x-auth-token", + "vendor_password", + "VENDOR_PASSWORD", + ]) { + expect(isSecretFieldName(name), `${name} MUST stay redacted`).toBe(true); + } + }); + + // --------------------------------------------------------------------------- + // PINNED KNOWN GAP — the end-anchor hole. + // + // SECRET_FIELD_PATTERN is END-ANCHORED (`/^(.*token|.*secret|…)$/i`), so a + // credential name with ANY suffix after the keyword is not flagged BY NAME: + // + // AWS_BEARER_TOKEN_BEDROCK → false (a live provider credential) + // SECRETS_MASTER_KEY → false (the master encryption key) + // SECRET_KEY_OLD → false + // MY_TOKEN_V2 → false + // + // These are NOT necessarily leaks: `looksLikeSecretValue` is the other half of + // the defense and catches a real high-entropy credential by VALUE. But the + // field-name half has a real hole, and widening the pattern to `.*token.*` is + // exactly the "obvious security fix is often wrong" trap — it would sweep in + // `max_tokens_to_sample`, `token_count_by_model`, `tokenizer_config`, … so the + // over-match exception set would have to grow unboundedly. + // + // This test pins the CURRENT behaviour deliberately so the gap is visible and a + // half-fix fails loudly. Closing it properly needs a keyword-BOUNDARY matcher + // (split on separators/camel-case, then test each segment) rather than a suffix + // matcher — widening the pattern to `.*token.*` would sweep in + // `max_tokens_to_sample`, `token_count_by_model`, `tokenizer_config`, …. + // --------------------------------------------------------------------------- + it("PINS the end-anchor gap: a credential name with a suffix is not flagged BY NAME", () => { + for (const name of [ + "AWS_BEARER_TOKEN_BEDROCK", + "SECRETS_MASTER_KEY", + "SECRET_KEY_OLD", + "MY_TOKEN_V2", + ]) { + expect( + isSecretFieldName(name), + `${name}: if this now returns true, the end-anchor gap was addressed — move it into the positive matrix above`, + ).toBe(false); + } + }); + it("flags pattern-matched secret field names (botToken, appSecret, …)", () => { for (const name of [ "botToken", diff --git a/packages/core/src/security/secret-detection.ts b/packages/core/src/security/secret-detection.ts index a4b54092a3..3e9e779311 100644 --- a/packages/core/src/security/secret-detection.ts +++ b/packages/core/src/security/secret-detection.ts @@ -230,12 +230,74 @@ const SECRET_FIELD_NAMES: ReadonlySet = new Set([ "api-key", ]); +/** + * Exact (lowercased) field names the `.*token` WILDCARD in + * {@link SECRET_FIELD_PATTERN} over-matches but which are NOT credentials. + * + * ⚠ EXACT names only — never a pattern. A pattern exception here would be a leak + * class (`.*_token` would admit `api_token`, `bot_token`, `refresh_token`). Each + * entry is a name whose semantics are a COUNT, a LIMIT, or a SELECTOR, and every + * one is pinned by the negative matrix in `secret-detection.test.ts` alongside the + * credential names that must keep matching. + * + * Why this exists: an MCP tool argument named `range_token`, holding a plain enum + * value, matched `.*token`, so LCD + session persistence rewrote it to the + * redaction placeholder. The model then read that placeholder back out of its own + * replay context and sent the literal string "[REDACTED]" to the server, which + * rejected it (`invalid_enum_value received:"[REDACTED]"`). Redacting a + * non-secret is not a safe default — it silently corrupts both the tool contract + * and the model's own history. + * + * Adding an entry is a security review: it must be a name that CANNOT carry a + * credential in any integration, not merely one that does not today. + */ +const NON_SECRET_TOKEN_FIELD_NAMES: ReadonlySet = new Set([ + // Range/window SELECTORS (an enum or date-range shorthand, never a credential). + "range_token", + "rangetoken", + // Token COUNTS + LIMITS (provider accounting + request knobs). + "tokens", + "max_tokens", + "maxtokens", + "min_tokens", + "mintokens", + "num_tokens", + "numtokens", + "token_count", + "tokencount", + "token_limit", + "tokenlimit", + "input_tokens", + "inputtokens", + "output_tokens", + "outputtokens", + "total_tokens", + "totaltokens", + "prompt_tokens", + "prompttokens", + "completion_tokens", + "completiontokens", + "cache_read_input_tokens", + "cache_creation_input_tokens", + "cached_tokens", + "cachedtokens", + // The tokenizer itself is a component name, never a credential. + "tokenizer", +]); + /** * True if a FIELD NAME implies a secret: - * `SECRET_FIELD_PATTERN` ∪ the header set above, case-insensitive. + * `SECRET_FIELD_PATTERN` ∪ the header set above, case-insensitive, MINUS the + * audited {@link NON_SECRET_TOKEN_FIELD_NAMES} over-match exceptions. + * + * Order matters: the exception set is consulted FIRST and only ever REMOVES a + * `.*token` wildcard match — an exact header name (`x-auth-token`) is not in the + * exception set, so the header superset is unaffected. */ export function isSecretFieldName(name: string): boolean { - return SECRET_FIELD_PATTERN.test(name) || SECRET_FIELD_NAMES.has(name.toLowerCase()); + const lower = name.toLowerCase(); + if (NON_SECRET_TOKEN_FIELD_NAMES.has(lower)) return false; + return SECRET_FIELD_PATTERN.test(name) || SECRET_FIELD_NAMES.has(lower); } // ── Env-ref exemption (reuses the EXPORTED env-substitution patterns) ── diff --git a/packages/daemon/src/api/env-handlers.ts b/packages/daemon/src/api/env-handlers.ts index 82d25a5957..f5b3b86872 100644 --- a/packages/daemon/src/api/env-handlers.ts +++ b/packages/daemon/src/api/env-handlers.ts @@ -176,10 +176,20 @@ export function createEnvHandlers(deps: EnvHandlerDeps): Record = {}): IncidentSignals { + return { + contextBudget: { + windowTokens: 1_000_000, + rawContextWindowTokens: 1_000_000, + windowCapSource: "none", + systemTokens: 54_477, + freshTailTokens: 712, + budgetedHistoryTokens: 32_551, + keptCount: 76, + assembledInputTokens: 87_740, + outputHeadroom: 3_840, + verdict: "fits", + freshTailSteps: 6, + freshTailStepsConfigured: 8, + ...over, + }, + } as unknown as IncidentSignals; +} + +describe("freshTailClampedVerdict", () => { + it("fires when the effective step bound is below the configured freshTailTurns", () => { + const v = freshTailClampedVerdict(signals()); + expect(v).not.toBeNull(); + expect(v!.code).toBe("fresh_tail_clamped"); + }); + + it("names BOTH numbers and the knob (so the operator does not have to grep DEBUG)", () => { + const v = freshTailClampedVerdict(signals())!; + expect(v.detail).toContain("contextEngine.freshTailTurns"); + expect(v.detail).toContain("6"); + expect(v.detail).toContain("8"); + }); + + it("states that the token budget was NOT the constraint (the 'fits' trap)", () => { + const v = freshTailClampedVerdict(signals())!; + // 87,740 / 1,000,000 = 9% — the window was nearly empty while the request slid out. + expect(v.detail).toMatch(/9% of the window/); + expect(v.detail).toMatch(/NOT the constraint/i); + }); + + it("stays silent when the configured value is honored", () => { + expect(freshTailClampedVerdict(signals({ freshTailSteps: 8 }))).toBeNull(); + expect(freshTailClampedVerdict(signals({ freshTailSteps: 12 }))).toBeNull(); + }); + + it("stays silent on a trajectory that predates the signal (both fields absent)", () => { + expect( + freshTailClampedVerdict( + signals({ freshTailSteps: undefined, freshTailStepsConfigured: undefined }), + ), + ).toBeNull(); + }); + + it("stays silent when there is no budget evidence at all", () => { + expect(freshTailClampedVerdict({} as unknown as IncidentSignals)).toBeNull(); + }); +}); diff --git a/packages/daemon/src/api/obs-handlers/obs-explain-fresh-tail-verdict.ts b/packages/daemon/src/api/obs-handlers/obs-explain-fresh-tail-verdict.ts new file mode 100644 index 0000000000..5a2ec7851e --- /dev/null +++ b/packages/daemon/src/api/obs-handlers/obs-explain-fresh-tail-verdict.ts @@ -0,0 +1,65 @@ +// SPDX-License-Identifier: Apache-2.0 +/** + * The `fresh_tail_clamped` verdict — the operator's verbatim-tail knob was + * silently overridden. + * + * Sibling of the spend/subagent verdicts (subdir line cap). Pure + deterministic: + * same signals in → same verdict out, no LLM. + * + * WHY this exists (comis-moshe 2026-07-26): the verbatim fresh tail is bounded by + * STEP COUNT, and the clamp reduced the operator's configured value to a smaller + * one on every finite window. On a turn with more tool round-trips than the + * effective bound, the user's ORIGINATING request slid out of verbatim context — + * and the agent then apologized for work the user had explicitly asked for. Every + * numeric lens read healthy at that moment: `verdict:"fits"`, `droppedCount:0`, + * 87,740 of 1,000,000 tokens in use. Nothing in `explain` named the slide, so the + * diagnosis required a DEBUG-only `lcd fresh tail sliced verbatim` log line. + * + * @module + */ + +import type { IncidentSignals, IncidentReport } from "@comis/core"; + +/** + * Stamp the verdict when the EFFECTIVE verbatim-tail step bound is below the + * operator-CONFIGURED value. + * + * Ranked LAST in the registry: a real tool/breaker/terminal cause is upstream of + * a context-shaping advisory and must out-rank it. It fires on the otherwise + * clean session — exactly the case that previously produced no verdict at all. + * + * @param s - the folded incident signals. + * @returns the root-cause verdict, or `null` when the knob was honored (or the + * trajectory predates the signal, in which case both fields are absent). + */ +export function freshTailClampedVerdict( + s: IncidentSignals, +): IncidentReport["likelyRootCause"] | null { + const b = s.contextBudget; + if (b === undefined) return null; + const effective = b.freshTailSteps; + const configured = b.freshTailStepsConfigured; + if (effective === undefined || configured === undefined) return null; + if (effective >= configured) return null; + + const windowUsedPct = + b.windowTokens > 0 ? Math.round((b.assembledInputTokens / b.windowTokens) * 100) : 0; + + return { + code: "fresh_tail_clamped", + detail: + `the verbatim fresh tail kept only ${String(effective)} trailing steps, but ` + + `contextEngine.freshTailTurns is configured as ${String(configured)} — the operator's value was ` + + `clamped. A turn with more than ${String(effective)} tool round-trips slides the user's own ` + + "request out of verbatim context, so the model can answer as though it was never asked. " + + `This turn used ${String(b.assembledInputTokens)} of ${String(b.windowTokens)} tokens ` + + `(${String(windowUsedPct)}% of the window), so the eviction budget was NOT the constraint — ` + + "the step bound was.", + suggestedNextSteps: [ + `compare the effective bound (${String(effective)}) against contextEngine.freshTailTurns ` + + `(${String(configured)}) — a persistent gap means the clamp, not the config, is deciding`, + "check whether the turn's tool round-trips exceeded the effective bound (the trajectory's " + + "tool.call records between the user's prompt.submitted and the reply)", + ], + }; +} diff --git a/packages/daemon/src/api/obs-handlers/obs-explain-heuristics.ts b/packages/daemon/src/api/obs-handlers/obs-explain-heuristics.ts index f70a420ae4..83d18e90ea 100644 --- a/packages/daemon/src/api/obs-handlers/obs-explain-heuristics.ts +++ b/packages/daemon/src/api/obs-handlers/obs-explain-heuristics.ts @@ -75,6 +75,7 @@ import { import { learnedSkillFailingVerdict, synthesisAbstainedVerdict } from "./obs-explain-learning-verdicts.js"; import { spendExceededVerdict } from "./obs-explain-spend-verdict.js"; // NAMED spend verdict (sibling — subdir cap) import { subagentStuckKilledVerdict } from "./obs-explain-subagent-killed-verdict.js"; // health-monitor-killed sub-agent (sibling — subdir cap) +import { freshTailClampedVerdict } from "./obs-explain-fresh-tail-verdict.js"; // verbatim-tail clamp advisory (sibling — subdir cap) import { backgroundPendingVerdict, backgroundRecoveryVerdict, @@ -569,6 +570,13 @@ export const HEURISTICS: ReadonlyArray<(s: IncidentSignals) => RootCause | null> ], }; }, + + // N) fresh_tail_clamped — DEAD LAST. A context-SHAPING advisory, not a + // terminal cause: every acute verdict above out-ranks it. It fires on the + // session that previously produced NO verdict at all — a "clean" turn whose + // originating request quietly slid out of the verbatim tail because the + // step bound (not the token budget) evicted it. Sibling module (subdir cap). + freshTailClampedVerdict, ]; /** Run the ordered registry; first non-null `RootCause` wins, else `null` (clean session). */ diff --git a/packages/daemon/src/api/obs-handlers/obs-explain.test.ts b/packages/daemon/src/api/obs-handlers/obs-explain.test.ts index 7d399b513f..fa02d9a591 100644 --- a/packages/daemon/src/api/obs-handlers/obs-explain.test.ts +++ b/packages/daemon/src/api/obs-handlers/obs-explain.test.ts @@ -691,6 +691,69 @@ describe("bindObsExplainHandlers", () => { expect(r.likelyRootCause?.detail).toMatch(/lossy|candidate|partial/i); }); + it("an unresolvable sessionKey whose ranker surfaces DIFFERENT real keys yields session_not_found (not a clean-looking empty report)", async () => { + // The live friction (comis-moshe 2026-07-26): `explain + // "telegram:~peer~"` CONTAINS a ':' so the CLI routes it as a + // {sessionKey} — never a traceId — so `refResolutionMissed` (traceId/rootRunId + // only) stayed false and the report came back ALL-ZERO with + // likelyRootCause:null, truncations:[] … while coverage.candidateSessionKeys + // held the correct full key. A silent wrong answer: indistinguishable from a + // healthy zero-activity session. + // + // The evidence that the ref did not resolve is the ranker surfacing OTHER + // real keys: a genuinely-real-but-empty session ranks itself (or ranks + // nothing), never a set that excludes it. + const realKey = + "default:agent:default:platform_o9UpY:telegram:peer:platform_o9UpY"; + const reader: IncidentSourceReader = { + readSessionRecords: async () => [], + readCacheTraceRecords: async () => [], + readSessionMetadata: async () => null, + readDiagnosticsRollup: async () => null, + listCandidateSessionKeys: async () => [realKey], + }; + const dataDir = fs.mkdtempSync(path.join(os.tmpdir(), "obs-explain-lossy-sk-")); + const handlers = bindObsExplainHandlers(makeDeps({ dataDir, incidentReader: reader })); + const r = (await handlers["obs.explain"]!({ + sessionKey: "telegram:platform_o9UpY~peer~platform_o9UpY", + _trustLevel: "admin", + })) as IncidentReport; + + expect(r.likelyRootCause).not.toBeNull(); + expect(r.likelyRootCause?.code).toBe("session_not_found"); + expect(r.likelyRootCause?.detail).toMatch(/lossy|candidate|partial/i); + // …and it names the field that missed, so the ledger reads honestly. + expect( + r.truncations.some( + (t) => t.field === "sessionKey" && /not\s*found|unresolv/i.test(t.reason), + ), + ).toBe(true); + // The operator still gets the key to copy. + expect(r.coverage?.candidateSessionKeys).toContain(realKey); + }); + + it("a REAL sessionKey the ranker ALSO returns stays a clean empty report (zero-telemetry, not not-found)", async () => { + // The precision guard for the rule above: when the ranker returns the very + // key that was requested, the session exists and simply has no telemetry — + // it must keep the null rootCause (no false not-found). + const realKey = "default:clean:clean:peer:clean"; + const reader: IncidentSourceReader = { + readSessionRecords: async () => [], + readCacheTraceRecords: async () => [], + readSessionMetadata: async () => null, + readDiagnosticsRollup: async () => null, + listCandidateSessionKeys: async () => [realKey], + }; + const dataDir = fs.mkdtempSync(path.join(os.tmpdir(), "obs-explain-real-empty-")); + const handlers = bindObsExplainHandlers(makeDeps({ dataDir, incidentReader: reader })); + const r = (await handlers["obs.explain"]!({ + sessionKey: realKey, + _trustLevel: "admin", + })) as IncidentReport; + expect(r.likelyRootCause).toBeNull(); + expect(r.truncations.some((t) => t.field === "sessionKey")).toBe(false); + }); + it("an EMPTY but RESOLVED session keeps the no-throw, null-rootCause behavior (only the UNRESOLVED case is marked)", async () => { // A real sessionKey that simply has no telemetry on disk must NOT be tagged // session_not_found — it resolved fine; it is just empty. Only the diff --git a/packages/daemon/src/api/obs-handlers/obs-explain.ts b/packages/daemon/src/api/obs-handlers/obs-explain.ts index acedcef979..8622e5885c 100644 --- a/packages/daemon/src/api/obs-handlers/obs-explain.ts +++ b/packages/daemon/src/api/obs-handlers/obs-explain.ts @@ -356,6 +356,27 @@ export async function assembleIncidentReportFromSources( records.length === 0 && metadata === null && reader.listCandidateSessionKeys ? await reader.listCandidateSessionKeys(candidateSeed) : []; + // A NON-EMPTY `sessionKey` that resolved nothing is ALSO unresolved — and it + // never reached the traceId/rootRunId arms above, so `refResolutionMissed` + // stayed false and the empty report masqueraded as a healthy zero-activity + // session. Live friction (comis-moshe 2026-07-26): `explain + // "telegram:~peer~"` CONTAINS a ':' so the CLI routes it as a + // sessionKey; the report came back all-zero with likelyRootCause:null while + // coverage.candidateSessionKeys already held the correct full key. + // + // The DISCRIMINATOR is the ranker: a session that genuinely exists but has no + // telemetry ranks ITSELF; a key that does not exist ranks only OTHER keys. So + // "ranker returned candidates, none of which is the requested key" is positive + // evidence the ref missed — while an empty candidate list (or one containing + // the request) keeps the honest clean-empty report. + const sessionKeyRefMissed = + params.sessionKey !== undefined && + records.length === 0 && + metadata === null && + candidateSessionKeys.length > 0 && + !candidateSessionKeys.includes(params.sessionKey); + const anyRefMissed = refResolutionMissed || sessionKeyRefMissed; + const missedField = sessionKeyRefMissed ? "sessionKey" : missedRefField; // Pass the trajectory READ count (records.length) so coverage.trajectory // reflects what the reader actually READ — the meta-observability point: a // "reader read nothing" bug surfaces as coverage.trajectory.records:0 @@ -393,7 +414,7 @@ export async function assembleIncidentReportFromSources( report.breakerTimeline.length === 0 && report.offloads.length === 0 && Object.keys(report.toolStats).length === 0; - if (refResolutionMissed && reportIsEmpty) { + if (anyRefMissed && reportIsEmpty) { // An honest not-found verdict + ledger note so the empty report // does not masquerade as a healthy zero-activity session. The bound pass // preserves both (it seeds truncations[] from the report and never @@ -408,7 +429,7 @@ export async function assembleIncidentReportFromSources( candidateSessionKeys.length > 0 ? { code: "session_not_found", - detail: `${missedRefField} did not resolve — it looks like a lossy/partial sessionKey (e.g. a bare chatId or the "~peer~" trajectory-filename form), which routes to a traceId lookup and misses. See coverage.candidateSessionKeys for the closest real keys.`, + detail: `${missedField} did not resolve — it looks like a lossy/partial sessionKey (e.g. a bare chatId or the "~peer~" trajectory-filename form), which matches no session on disk. See coverage.candidateSessionKeys for the closest real keys.`, suggestedNextSteps: [ `re-run with one of the candidate session keys (coverage.candidateSessionKeys), e.g. "${candidateSessionKeys[0]}"`, "or pass the traceId from the trajectory's model.completed record", @@ -416,16 +437,16 @@ export async function assembleIncidentReportFromSources( } : { code: "session_not_found", - detail: `${missedRefField} did not resolve to any session in the index (today/yesterday). Either the reference is wrong or older than the 2-day resolution horizon, OR the turn aborted BEFORE it recorded a session — a pre-execution fail-closed (e.g. unresolved conversation authority, a rejected identity, or tool assembly) never reaches the index, so a real failure can look like a missing reference`, + detail: `${missedField} did not resolve to any session in the index (today/yesterday). Either the reference is wrong or older than the 2-day resolution horizon, OR the turn aborted BEFORE it recorded a session — a pre-execution fail-closed (e.g. unresolved conversation authority, a rejected identity, or tool assembly) never reaches the index, so a real failure can look like a missing reference`, suggestedNextSteps: [ - `verify the ${missedRefField}, or query by sessionKey directly`, + `verify the ${missedField}, or query by sessionKey directly`, "confirm the session ended within the last two days (the session-index lookup window)", `if the trigger is known to have fired, treat this as a possible pre-execution abort: grep the daemon log for this ${missedRefField} — a turn that failed before recording emits an ERROR carrying its step and errorKind but no session`, ], }; report.truncations.push({ - field: missedRefField, - reason: `${missedRefField} not found in session index (today/yesterday) — empty report is unresolved, not a clean session`, + field: missedField, + reason: `${missedField} not found in session index (today/yesterday) — empty report is unresolved, not a clean session`, }); } else { // Thread the mapped terminal endReason (the NAMED degradation cause) diff --git a/packages/daemon/src/api/obs-handlers/system-health.test.ts b/packages/daemon/src/api/obs-handlers/system-health.test.ts index 2f55c29d2e..09d11e20b2 100644 --- a/packages/daemon/src/api/obs-handlers/system-health.test.ts +++ b/packages/daemon/src/api/obs-handlers/system-health.test.ts @@ -345,6 +345,52 @@ describe("assembleSystemHealthReport (4-source read fan-in)", () => { expect(report.sessions.degraded).toBe(4); // the raw flag is unchanged (invariant) }); + // LIVE INCIDENT (comis-moshe 2026-07-26): the TABLE view printed + // "Sessions: 1 (0 hard-degraded, 0%; +1 delivered-with-tool-errors)" while the + // JSON of the SAME report said degraded:1, degradedRate:1. Two renderings of one + // report disagreed because the CLI derived the hard split locally and the JSON + // shipped only the raw flag — so a JSON consumer read 100% degraded on a system + // the human view called healthy. The split belongs in the report. + it("carries the HARD degraded split in the report so the JSON and the table cannot disagree", async () => { + const now = systemNowMs(); + const store = makeStore(); + for (let i = 0; i < 3; i++) { + store.insertDiagnostic({ + timestamp: now - 1_000 - i, category: "session_summary", severity: "warning", sessionKey: `d${i}`, + message: "session:summary", + details: summaryDetails({ degraded: true, endReason: "completed_with_tool_errors" }), + }); + } + store.insertDiagnostic({ + timestamp: now - 400, category: "session_summary", severity: "warning", sessionKey: "hard", + message: "session:summary", + details: summaryDetails({ degraded: true, endReason: "context_exhausted" }), + }); + const report = await assembleSystemHealthReport({ obsStore: store, dataDir: makeDataDirWithActivity(), clock: createFakeClock(now) }, 24); + + // The raw flag is preserved (invariant) … + expect(report.sessions.degraded).toBe(4); + expect(report.sessions.deliveredWithToolErrors).toBe(3); + // … and the HARD split the findings actually fire on is now IN the report. + expect(report.sessions.hardDegraded).toBe(1); + expect(report.sessions.hardDegradedRate).toBeCloseTo(0.25, 5); + }); + + it("reports a zero hard-degraded rate when every degraded session still delivered", async () => { + const now = systemNowMs(); + const store = makeStore(); + store.insertDiagnostic({ + timestamp: now - 1_000, category: "session_summary", severity: "warning", sessionKey: "only", + message: "session:summary", + details: summaryDetails({ degraded: true, endReason: "completed_with_tool_errors" }), + }); + const report = await assembleSystemHealthReport({ obsStore: store, dataDir: makeDataDirWithActivity(), clock: createFakeClock(now) }, 24); + // This is the EXACT live shape: 1 session, degraded:1, but 0 hard. + expect(report.sessions.degraded).toBe(1); + expect(report.sessions.hardDegraded).toBe(0); + expect(report.sessions.hardDegradedRate).toBe(0); + }); + it("STILL root-causes to acute degradation on a genuine HARD failure amid delivered-with-tool-errors", async () => { const now = systemNowMs(); const store = makeStore(); diff --git a/packages/daemon/src/api/obs-handlers/system-health.ts b/packages/daemon/src/api/obs-handlers/system-health.ts index eac2fe41d0..9082525804 100644 --- a/packages/daemon/src/api/obs-handlers/system-health.ts +++ b/packages/daemon/src/api/obs-handlers/system-health.ts @@ -500,10 +500,22 @@ export async function assembleSystemHealthReport( ? ["no sessions in the window — widen `--since` or confirm the daemon is recording session summaries"] : ["run `comis explain ` on the worst session for the per-session post-mortem"]; + const hardDegradedCount = Math.max(0, degraded - system.deliveredWithToolErrorsCount); + return { schemaVersion: 1, windowHours, - sessions: { total: system.sessionCount, degraded, degradedRate: system.degradedRate, deliveredWithToolErrors: system.deliveredWithToolErrorsCount }, + sessions: { + total: system.sessionCount, + degraded, + degradedRate: system.degradedRate, + deliveredWithToolErrors: system.deliveredWithToolErrorsCount, + // The HARD split the findings fire on, computed ONCE here so every + // renderer agrees (the CLI used to derive it locally and disagree with + // this report's own JSON). + hardDegraded: hardDegradedCount, + hardDegradedRate: system.sessionCount > 0 ? hardDegradedCount / system.sessionCount : 0, + }, topErrorKinds, // The system-level degradation detector: degraded counts by named // endReason cause, computed by reduceSystemWindow from the per-session rows @@ -520,7 +532,7 @@ export async function assembleSystemHealthReport( // figure as `activity.tokenTotal` (a single source of truth — no second // aggregate); consumers cross-reference `coverage` before trusting a 0. The // `comis system-health` table render drops the misleading "· 0 tok" in that case. - cost: { costUsd: system.costUsd, totalTokens: activity.tokenTotal, offSessionUsd }, + cost: { costUsd: system.costUsd, totalTokens: activity.tokenTotal, tokenBasis: "input+output" as const, offSessionUsd }, activity: { activeAgents: activity.activeAgents, activeChannels: activity.activeChannels, diff --git a/packages/daemon/src/daemon.ts b/packages/daemon/src/daemon.ts index bf49f87f7b..a77413b263 100644 --- a/packages/daemon/src/daemon.ts +++ b/packages/daemon/src/daemon.ts @@ -8,7 +8,8 @@ * reusable wiring lives under `./wiring/`. * * @module - */ + */import { assertProactiveFailureIsSupported, proactiveNotArmedLogFields, PROACTIVE_NOT_ARMED_MSG, EMPTY_PROACTIVE_HANDLES } from "./wiring/proactive-degrade.js"; + import { bootstrap, loadEnvFile, @@ -2445,18 +2446,16 @@ async function bootChannels(boot: BootContext): Promise { // Seed the four canonical small-model DAG templates into the named-graph store. Idempotent (INSERT-OR-IGNORE in the seeder), so operator-customized templates survive restarts and re-running on every boot is safe. seedDefaultDagTemplates(namedGraphStore); - // 6.7. Bind every proactive dependency, then arm heartbeat and cron together. + // 6.7. Bind proactive deps, then arm heartbeat + cron. Autonomy-disabled is + // SUPPORTED and must not crash-loop the daemon (wiring/proactive-degrade.ts). const proactive = await setupProactiveSchedulers({ - runtime: handle, - adaptersByType, - deliveryService, + runtime: handle, adaptersByType, deliveryService, schedulerCorePortBindings: handle.schedulerCorePortBindings, }); - if (!proactive.ok) { - throw new Error(`Proactive scheduler activation failed: ${proactive.error.message}`); - } - handle.bindTaskMaintenanceRuntime(proactive.value); - const { heartbeatRunner, duplicateDetector, coordinator: heartbeatCoordinator } = proactive.value; + assertProactiveFailureIsSupported(proactive); + if (proactive.ok) handle.bindTaskMaintenanceRuntime(proactive.value); + else daemonLogger.error(proactiveNotArmedLogFields(), PROACTIVE_NOT_ARMED_MSG); + const { heartbeatRunner, duplicateDetector, coordinator: heartbeatCoordinator } = proactive.ok ? proactive.value : EMPTY_PROACTIVE_HANDLES; // 6.7.0.2. Agent management runtime state const suspendedAgents = new Set(); const modelCatalog = createModelCatalog(); @@ -2479,7 +2478,9 @@ async function bootChannels(boot: BootContext): Promise { videoGenProvider, videoGenRateLimiter, videoGenConfig, persistVideo, videoGenCostLimiter, videoJobStore, videoPoller, preprocessMessageText, getCapabilityPortForAgent, heartbeatRunner, duplicateDetector, heartbeatCoordinator, - proactiveSchedulers: proactive.value, + // Absent when autonomy is disabled for every agent (the honest-degrade boot + // above) — consumers must treat the proactive surface as optional. + ...(proactive.ok ? { proactiveSchedulers: proactive.value } : {}), nodeTypeRegistry, graphCoordinator, namedGraphStore, suspendedAgents, modelCatalog, channelConfig, promptTimeoutTimestamps, // Teardown handles surfaced for ShutdownDeps wiring. diff --git a/packages/daemon/src/wiring/proactive-degrade.ts b/packages/daemon/src/wiring/proactive-degrade.ts new file mode 100644 index 0000000000..632fd10333 --- /dev/null +++ b/packages/daemon/src/wiring/proactive-degrade.ts @@ -0,0 +1,86 @@ +// SPDX-License-Identifier: Apache-2.0 +// @allow-throw: boot-path wiring guard — the throw aborts daemon startup for a genuine composition-root regression (same contract as the daemon.ts call site it was extracted from). +/** + * Honest degradation when proactive schedulers cannot be armed. + * + * `constructCapabilityLayer` returns `capEndpointHandle: undefined` BY DESIGN + * when no agent has autonomy enabled, and `setupProactiveSchedulers` treats that + * handle as mandatory. The daemon used to throw on the failed Result — so a fully + * supported config completed its entire boot (channels registered, adapter + * polling) and then exited 1, forever. `systemctl is-active` read `active` + * throughout while the box served nothing. + * + * Reachability is the severity: an omitted `autonomy` block defaults to ENABLED, + * but writing any sub-key without `enabled: true` — e.g. the documented + * `autonomy.durability` — resolves to DISABLED. One documented knob bricked it. + * + * @module + */ + +/** The failure shape a failed `setupProactiveSchedulers` returns. */ +interface ProactiveSetupError { + readonly code: string; + readonly message: string; +} + +/** + * True when the ONLY unmet proactive-scheduler dependency is the capability + * endpoint — i.e. the supported autonomy-disabled configuration, not a + * composition-root regression. + * + * @param error - the failed setup Result's error. + * @returns whether the daemon may boot without the proactive surface. + */ +export function isAutonomyDisabledProactiveMiss(error: ProactiveSetupError): boolean { + return ( + error.code === "dependency_unavailable" + && /capEndpointHandle \(1 of 11/.test(error.message) + ); +} + +/** + * Structured fields for the boot ERROR that says what is off and which knob + * turns it back on. Never a silent loss. + * + * @returns content-free log fields. + */ +export function proactiveNotArmedLogFields(): Record { + return { + module: "daemon", + submodule: "setup-proactive-schedulers", + errorKind: "config" as const, + hint: + "Cron jobs and the heartbeat are NOT armed for this boot. The capability endpoint " + + "is only built when at least one agent has autonomy enabled, and no agent does. " + + "Set `agents..autonomy.enabled: true` to arm them. NOTE: writing any " + + "`autonomy.*` sub-key (e.g. `autonomy.durability`) WITHOUT `autonomy.enabled: true` " + + "resolves to DISABLED — an omitted `autonomy` block defaults to enabled, so adding " + + "a sub-key can silently turn the whole surface off.", + }; +} + +/** The boot ERROR message for {@link proactiveNotArmedLogFields}. */ +export const PROACTIVE_NOT_ARMED_MSG = + "Proactive schedulers not armed: autonomy is disabled for every agent"; + +/** + * Abort boot unless a failed proactive-scheduler setup is the SUPPORTED + * autonomy-disabled case. + * + * @param proactive - the `setupProactiveSchedulers` Result. + * @throws when the failure is a genuine composition-root regression. + */ +export function assertProactiveFailureIsSupported( + proactive: { ok: true } | { ok: false; error: ProactiveSetupError }, +): void { + if (proactive.ok) return; + if (isAutonomyDisabledProactiveMiss(proactive.error)) return; + throw new Error(`Proactive scheduler activation failed: ${proactive.error.message}`); +} + +/** Handle placeholders for the degraded (schedulers-not-armed) boot. */ +export const EMPTY_PROACTIVE_HANDLES = { + heartbeatRunner: undefined, + duplicateDetector: undefined, + coordinator: undefined, +} as const; diff --git a/packages/daemon/src/wiring/setup-observability.test.ts b/packages/daemon/src/wiring/setup-observability.test.ts index 2489ff4e0c..1d63fb7658 100644 --- a/packages/daemon/src/wiring/setup-observability.test.ts +++ b/packages/daemon/src/wiring/setup-observability.test.ts @@ -258,10 +258,20 @@ describe("setupObservability", () => { // ------------------------------------------------------------------------- // ------------------------------------------------------------------------- - // 7. Subscribes to observability:cache_break event + // 7-8. Cache-break logging is SINGLE-SOURCE (the detector owns it). + // + // These two tests used to assert that setupObservability subscribed to + // `observability:cache_break` and logged it at INFO. That subscription was a + // pure PROJECTION of an event `cache-state.ts` already logs under the SAME + // `msg`, so every break produced TWO identical INFO lines and every lens that + // counts by message double-counted (a 9-break session read as 18, inflating the + // `cache_prefix_churn` system finding). The wiring no longer re-logs; the + // detector's single line carries all the fields this one used to add. The + // single-source invariant is pinned in + // packages/agent/src/executor/cache-detection/cache-state.test.ts. // ------------------------------------------------------------------------- - it("subscribes to observability:cache_break event", async () => { + it("does NOT re-log cache_break (the detector is the single source of the count)", async () => { const eventBus = createMockEventBus(); const setupObservability = await getSetupObservability(); const mockLogger = { info: vi.fn() }; @@ -272,68 +282,22 @@ describe("setupObservability", () => { logger: mockLogger, }); - const handlers = eventBus._handlers.get("observability:cache_break"); - expect(handlers).toBeDefined(); - expect(handlers!.length).toBeGreaterThan(0); - }); - - // ------------------------------------------------------------------------- - // 8. Emitting cache_break logs at INFO with structured fields - // ------------------------------------------------------------------------- - - it("logs cache_break event with structured fields at INFO level", async () => { - const eventBus = createMockEventBus(); - const setupObservability = await getSetupObservability(); - const mockLogger = { info: vi.fn() }; - - await setupObservability({ - eventBus: eventBus as any, - _createTokenTracker: mockCreateTokenTracker, - logger: mockLogger, - }); - - const payload = { + eventBus.emit("observability:cache_break", { provider: "anthropic", reason: "system_prompt_changed", tokenDrop: 5000, tokenDropRelative: 0.42, - previousCacheRead: 12000, - currentCacheRead: 7000, - callCount: 15, - changes: { - systemChanged: true, - toolsChanged: false, - metadataChanged: false, - modelChanged: false, - retentionChanged: false, - addedTools: [], - removedTools: [], - changedSchemaTools: [], - }, - toolsChanged: ["tool-a", "tool-b"], - ttlCategory: "medium", - agentId: "agent-test", - sessionKey: "session-test", - timestamp: Date.now(), - }; - - eventBus.emit("observability:cache_break", payload); - - expect(mockLogger.info).toHaveBeenCalledWith( - expect.objectContaining({ - provider: "anthropic", - reason: "system_prompt_changed", - tokenDrop: 5000, - tokenDropRelative: 0.42, - agentId: "agent-test", - sessionKey: "session-test", - ttlCategory: "medium", - toolsChanged: 2, - systemChanged: true, - modelChanged: false, - }), - "Cache break detected", + agentId: "default", + sessionKey: "s", + ttlCategory: "long", + toolsChanged: [], + changes: { systemChanged: true, modelChanged: false }, + } as never); + + const cacheBreakLogs = mockLogger.info.mock.calls.filter( + (c: unknown[]) => c[1] === "Cache break detected", ); + expect(cacheBreakLogs).toHaveLength(0); }); // ------------------------------------------------------------------------- diff --git a/packages/daemon/src/wiring/setup-observability.ts b/packages/daemon/src/wiring/setup-observability.ts index a1b221d2bd..a2ed2d4d9a 100644 --- a/packages/daemon/src/wiring/setup-observability.ts +++ b/packages/daemon/src/wiring/setup-observability.ts @@ -335,26 +335,11 @@ export async function setupObservability(deps: { } } - // Log cache break events for operational observability - if (deps.logger) { - eventBus.on("observability:cache_break", (payload) => { - deps.logger!.info( - { - provider: payload.provider, - reason: payload.reason, - tokenDrop: payload.tokenDrop, - tokenDropRelative: payload.tokenDropRelative, - agentId: payload.agentId, - sessionKey: payload.sessionKey, - ttlCategory: payload.ttlCategory, - toolsChanged: payload.toolsChanged.length, - systemChanged: payload.changes.systemChanged, - modelChanged: payload.changes.modelChanged, - }, - "Cache break detected", - ); - }); - } + // Cache-break events are logged ONCE, at the detector (`cache-state.ts`), which + // carries every field this wiring used to re-log. A second INFO with the same + // `msg` for one event made every counting lens double-count — a 9-break session + // read as 18 and inflated the `cache_prefix_churn` system finding accordingly. + // The diff-writer subscription below is the operational work that belongs here. // Persist cache break diagnostics to ~/.comis/cache-breaks/ if (deps.dataDir && deps.logger) { diff --git a/packages/daemon/src/wiring/setup-proactive-schedulers.test.ts b/packages/daemon/src/wiring/setup-proactive-schedulers.test.ts index 2f56627170..93b7e70ad4 100644 --- a/packages/daemon/src/wiring/setup-proactive-schedulers.test.ts +++ b/packages/daemon/src/wiring/setup-proactive-schedulers.test.ts @@ -65,14 +65,28 @@ describe("proactive scheduler composition", () => { }); it("rejects active startup before side effects when runtime dependencies are absent", async () => { - await expect(setupProactiveSchedulers(deps(runtime({ cronEnabled: true })))).resolves.toEqual({ - ok: false, - error: { - code: "dependency_unavailable", - errorKind: "precondition", - message: "Proactive scheduler dependency is unavailable", - }, - }); + const result = await setupProactiveSchedulers(deps(runtime({ cronEnabled: true }))); + expect(result.ok).toBe(false); + if (result.ok) throw new Error("expected err"); + expect(result.error.code).toBe("dependency_unavailable"); + expect(result.error.errorKind).toBe("precondition"); + }); + + // This guard collapses ELEVEN distinct runtime dependencies into one Result. + // When any is missing the daemon throws + // `FATAL: Proactive scheduler activation failed: ` and systemd + // restart-loops — so the message is the ONLY diagnostic an operator gets, and + // "Proactive scheduler dependency is unavailable" named none of the eleven. + // Observed live on a fresh install: a boot crash-loop with nothing to act on. + it("NAMES the missing dependencies so a boot crash-loop is diagnosable", async () => { + const result = await setupProactiveSchedulers(deps(runtime({ cronEnabled: true }))); + if (result.ok) throw new Error("expected err"); + // At least one concrete dependency identifier must appear in the message. + expect(result.error.message).toMatch( + /workspaceDirs|getExecutor|piSessionAdapters|assembleToolsForAgent|sharedLeaseManager|boundedAutonomyBudgetHolder|capEndpointHandle|cronRuntimeBinding|activateCronSchedulers|deactivateCronSchedulers|getAgentSchedulerSeed/, + ); + // …and it must still say what failed, for the FATAL prefix to read sensibly. + expect(result.error.message).toMatch(/proactive scheduler/i); }); it("binds and activates the complete task runtime instead of leaving an enabled capture-only path", () => { diff --git a/packages/daemon/src/wiring/setup-proactive-schedulers.ts b/packages/daemon/src/wiring/setup-proactive-schedulers.ts index 7da04183c7..9c8c5a7970 100644 --- a/packages/daemon/src/wiring/setup-proactive-schedulers.ts +++ b/packages/daemon/src/wiring/setup-proactive-schedulers.ts @@ -725,6 +725,17 @@ function resolveRequiredRuntime(runtime: ProactiveBootSlice): Result<{ deactivateCronSchedulers, getAgentSchedulerSeed, } = runtime; + // NAME the missing dependencies. This Result is the ONLY diagnostic an operator + // gets for this failure: `daemon.ts` turns it into + // `FATAL: Proactive scheduler activation failed: ` and exits, so + // systemd restart-loops on it. Collapsing eleven distinct wiring dependencies + // into one opaque sentence left a fresh install crash-looping with nothing to + // act on — the operator cannot tell a config problem from a composition-root + // regression. Identifiers only (no values) — these are wiring symbol names. + // + // The explicit `||` chain below is kept as the CONTROL-FLOW guard so TypeScript + // still narrows all eleven to non-undefined for the `ok(...)` return; the name + // list is built only on the failure branch. if ( workspaceDirs === undefined || getExecutor === undefined @@ -738,10 +749,31 @@ function resolveRequiredRuntime(runtime: ProactiveBootSlice): Result<{ || deactivateCronSchedulers === undefined || getAgentSchedulerSeed === undefined ) { + const missingDependencies = ( + [ + ["workspaceDirs", workspaceDirs], + ["getExecutor", getExecutor], + ["piSessionAdapters", piSessionAdapters], + ["assembleToolsForAgent", assembleToolsForAgent], + ["sharedLeaseManager", sharedLeaseManager], + ["boundedAutonomyBudgetHolder", boundedAutonomyBudgetHolder], + ["capEndpointHandle", capEndpointHandle], + ["cronRuntimeBinding", cronRuntimeBinding], + ["activateCronSchedulers", activateCronSchedulers], + ["deactivateCronSchedulers", deactivateCronSchedulers], + ["getAgentSchedulerSeed", getAgentSchedulerSeed], + ] as ReadonlyArray + ) + .filter(([, value]) => value === undefined) + .map(([name]) => name); return err({ code: "dependency_unavailable", errorKind: "precondition", - message: "Proactive scheduler dependency is unavailable", + message: + `Proactive scheduler dependency is unavailable: ${missingDependencies.join(", ")} ` + + `(${missingDependencies.length} of 11 runtime dependencies unset). ` + + "These are composition-root wiring handles, not config keys — an unset one means the " + + "daemon reached scheduler activation before that subsystem finished initializing.", }); } return ok({ diff --git a/packages/observability/src/trajectory/event-bus-bridge.test.ts b/packages/observability/src/trajectory/event-bus-bridge.test.ts index 4a8f9d3524..3d2ca6e575 100644 --- a/packages/observability/src/trajectory/event-bus-bridge.test.ts +++ b/packages/observability/src/trajectory/event-bus-bridge.test.ts @@ -4555,3 +4555,83 @@ describe("attachTrajectoryToEventBus direct recovery admission", () => { expect(recorder.calls).toHaveLength(0); }); }); + +describe("background-task dispatch redelivery is not a second failure", () => { + // Observed live: FOUR background tasks failed, but the trajectory carried + // FIFTY-FIVE `background_task.failed` records (17/15/13/10). + // `scheduleDispatchRetry` re-emits the terminal event on every backoff tick to + // re-drive dispatch — a legitimate mechanism — but each re-emit also wrote a new + // trajectory line, so one failure looked like seventeen and drowned the real + // signal in `explain`. The redelivery must re-drive dispatch WITHOUT duplicating + // the record (same posture as `background_task:notified`'s `trajectoryRecorded`). + const origin = { + turnScope: { + conversation: { + agentId: "agent-a", + sessionKey: "default:agent-a:echo:conversation-a:user_a", + }, + }, + } as never; + + function failedPayload(extra: Record) { + return { + agentId: "agent-a", + taskId: "task-a", + toolName: "mcp__vendor-mcp--vendor_activity_report", + error: "MCP error -32001: Request timed out", + durationMs: 120_000, + origin, + timestamp: 10, + ...extra, + }; + } + + it("records the FIRST terminal failure", () => { + const bus = makeBus(); + const recorder = createCaptureRecorder(); + attachTrajectoryToEventBus({ + eventBus: bus, + recorder, + ownerSessionKey: "default:agent-a:echo:conversation-a:user_a", + }); + bus.emit("background_task:failed", failedPayload({}) as never); + expect(recorder.calls).toHaveLength(1); + }); + + it("does NOT record a dispatch-retry redelivery of the same failure", () => { + const bus = makeBus(); + const recorder = createCaptureRecorder(); + attachTrajectoryToEventBus({ + eventBus: bus, + recorder, + ownerSessionKey: "default:agent-a:echo:conversation-a:user_a", + }); + bus.emit("background_task:failed", failedPayload({}) as never); + // …the backoff ticks that re-drive dispatch (live: up to 16 of them per task) + for (let i = 0; i < 16; i += 1) { + bus.emit("background_task:failed", failedPayload({ dispatchRedelivery: true }) as never); + } + expect(recorder.calls).toHaveLength(1); + }); + + it("does NOT record a dispatch-retry redelivery of a completion either", () => { + const bus = makeBus(); + const recorder = createCaptureRecorder(); + attachTrajectoryToEventBus({ + eventBus: bus, + recorder, + ownerSessionKey: "default:agent-a:echo:conversation-a:user_a", + }); + const completed = { + agentId: "agent-a", + taskId: "task-b", + toolName: "report", + durationMs: 5, + origin, + timestamp: 10, + }; + bus.emit("background_task:completed", completed as never); + bus.emit("background_task:completed", { ...completed, dispatchRedelivery: true } as never); + expect(recorder.calls).toHaveLength(1); + }); +}); diff --git a/packages/observability/src/trajectory/event-bus-bridge.ts b/packages/observability/src/trajectory/event-bus-bridge.ts index 6789f54e03..7eaf339767 100644 --- a/packages/observability/src/trajectory/event-bus-bridge.ts +++ b/packages/observability/src/trajectory/event-bus-bridge.ts @@ -599,6 +599,16 @@ export function attachTrajectoryToEventBus( ) { return; } + // A `scheduleDispatchRetry` backoff tick re-emits the TERMINAL event to + // re-drive dispatch; it is not a second failure/completion. Recording it + // turned one failure into up to seventeen trajectory lines and buried the + // real signal in `explain` (comis-moshe 2026-07-26: 55 records / 4 tasks). + if ( + (eventName === "background_task:failed" || eventName === "background_task:completed") + && (payload as { dispatchRedelivery?: boolean }).dispatchRedelivery === true + ) { + return; + } const data = translatePayload(eventName, payload); const trajectoryType = TRAJECTORY_BRIDGE_MAPPING[eventName]; recorder.recordEvent(trajectoryType, data); diff --git a/packages/observability/src/trajectory/translate-payload.ts b/packages/observability/src/trajectory/translate-payload.ts index e3891342de..300e99b0a0 100644 --- a/packages/observability/src/trajectory/translate-payload.ts +++ b/packages/observability/src/trajectory/translate-payload.ts @@ -767,6 +767,15 @@ export function translatePayload( windowCapSource: payload.windowCapSource, systemTokens: payload.systemTokens, freshTailTokens: payload.freshTailTokens, + // The verbatim-tail STEP bound: counts only. Effective-vs-configured is + // what makes a fresh-tail SLIDE visible in `explain` (a clean "fits" + // verdict hid it live — comis-moshe 2026-07-26). + ...(payload.freshTailSteps !== undefined + ? { freshTailSteps: payload.freshTailSteps } + : {}), + ...(payload.freshTailStepsConfigured !== undefined + ? { freshTailStepsConfigured: payload.freshTailStepsConfigured } + : {}), budgetedHistoryTokens: payload.budgetedHistoryTokens, keptCount: payload.keptCount, assembledInputTokens: payload.assembledInputTokens, diff --git a/packages/orchestrator/src/execution/activity-turn-coordinator.test.ts b/packages/orchestrator/src/execution/activity-turn-coordinator.test.ts index 2a6c2d271f..2a99803391 100644 --- a/packages/orchestrator/src/execution/activity-turn-coordinator.test.ts +++ b/packages/orchestrator/src/execution/activity-turn-coordinator.test.ts @@ -1216,3 +1216,53 @@ describe("createActivityTurnCoordinator — planStream subscription", () => { coord.dispose(); }); }); + +describe("a failure with no failed activity events never renders a naked errorKind", () => { + // LIVE INCIDENT (comis-moshe 2026-07-26): the user's chat received a bubble whose + // ENTIRE text was "❌ dependency". The turn finalized + // `{outcome:"failure", errorKind:"dependency", failedEventCount:0}` — the four + // MCP tools that actually failed had each been closed `status:"completed"` at + // background hand-off, so the card had no failed event to name and + // `failureLabel()` fell through to the bare errorKind token. This module's own + // comment says a kept "❌ {errorKind}" pill above a delivered answer must never + // happen; the guard only covered `outcome.kind === "success"`. + it("populates a truthful `reason` so the pill is not a bare token", async () => { + const clock = createFakeClock(5_000); + const { deps, renderer } = makeCoordinatorDeps({ clock }); + const coord = createActivityTurnCoordinator(deps); + coord.start(makeCtx()); + + await coord.finalize({ kind: "failure", errorKind: "dependency", failedEvents: [] }); + + expect(renderer.finalizeCalls.length).toBe(1); + const outcome = renderer.finalizeCalls[0].outcome as Extract< + typeof renderer.finalizeCalls[0]["outcome"], + { kind: "failure" } + >; + expect(outcome.kind).toBe("failure"); + expect(outcome.reason, "an unattributed failure must carry a reason").toBeDefined(); + expect(outcome.reason!.length).toBeGreaterThan(0); + coord.dispose(); + }); + + it("does NOT overwrite an explicit reason (a resource abort keeps its own wording)", async () => { + const clock = createFakeClock(5_000); + const { deps, renderer } = makeCoordinatorDeps({ clock }); + const coord = createActivityTurnCoordinator(deps); + coord.start(makeCtx()); + + await coord.finalize({ + kind: "failure", + errorKind: "resource", + failedEvents: [], + reason: "stopped — spend limit reached", + }); + + const outcome = renderer.finalizeCalls[0].outcome as Extract< + typeof renderer.finalizeCalls[0]["outcome"], + { kind: "failure" } + >; + expect(outcome.reason).toBe("stopped — spend limit reached"); + coord.dispose(); + }); +}); diff --git a/packages/orchestrator/src/execution/activity-turn-coordinator.ts b/packages/orchestrator/src/execution/activity-turn-coordinator.ts index d7a3b220ea..d5cb77bcb6 100644 --- a/packages/orchestrator/src/execution/activity-turn-coordinator.ts +++ b/packages/orchestrator/src/execution/activity-turn-coordinator.ts @@ -51,7 +51,7 @@ import type { ComisLogger, ErrorKind, } from "@comis/core"; -import { isNonEmptyEvents, redactValue, toSafeErrorLogString } from "@comis/core"; +import { isNonEmptyEvents, redactValue, toSafeErrorLogString, UNATTRIBUTED_FAILURE_REASON } from "@comis/core"; import type { Result } from "@comis/shared"; import { fromPromise, suppressError, tryCatch } from "@comis/shared"; import { randomUUID } from "node:crypto"; @@ -575,6 +575,22 @@ export function createActivityTurnCoordinator(deps: ActivityTurnCoordinatorDeps) } } + // (1b) An unattributed FAILURE must still read truthfully. A `failure` with + // zero failed events cannot name a tool, so `failureLabel()` renders the bare + // "❌ {errorKind}" — which reached a real user as a chat bubble whose entire + // text was "❌ dependency" (comis-moshe 2026-07-26: the four MCP tools that + // failed had each been closed status:"completed" at background hand-off, so + // failedEventCount was 0 on a genuine failure). Fill the closed-vocabulary + // reason the union already carries for exactly this purpose. An explicit + // reason (a resource abort's own wording) is never overwritten. + if ( + effective.kind === "failure" + && effective.failedEvents.length === 0 + && (effective.reason === undefined || effective.reason.length === 0) + ) { + effective = { ...effective, reason: UNATTRIBUTED_FAILURE_REASON }; + } + // Announce the EFFECTIVE terminal surface state (the user-visible pill's // fate is a pure function of this outcome + the strategy) so `explain` can // answer "what did the user's chat show this turn" from the trajectory — diff --git a/packages/skills/src/platform-tools/tools/gateway-tool.test.ts b/packages/skills/src/platform-tools/tools/gateway-tool.test.ts index 42dcc66aad..b63d978016 100644 --- a/packages/skills/src/platform-tools/tools/gateway-tool.test.ts +++ b/packages/skills/src/platform-tools/tools/gateway-tool.test.ts @@ -1,5 +1,6 @@ // SPDX-License-Identifier: Apache-2.0 import { describe, it, expect, vi, beforeEach } from "vitest"; +import * as fs from "node:fs"; // `isDocker` is consumed at gateway-tool module scope inside the restart // branch. Mock `@comis/core` with `importOriginal` so we only override @@ -1266,3 +1267,43 @@ describe("confirmationRequiredHint (approval foot-gun fix)", () => { expect(hint.toLowerCase()).toContain("does not perform it"); }); }); + +describe("the env_set placeholder hint must not cause credential re-transmission", () => { + // Observed live: a user was asked to paste their password into the chat a SECOND + // time. An approval-gated `env_set` needs a follow-up call carrying + // `_confirmed: true`, but the redaction pass rewrites the secret in the + // originating message between the two calls, and `scrubRedactedToolCalls` + // removes the earlier tool-call pair — so the value is genuinely gone from the + // agent's context by the time consent arrives. The old hint told the agent to + // "re-read the user's most recent message and call env_set again with the + // literal value", which can only be satisfied by asking for re-transmission. + function hint(): string { + const src = fs.readFileSync( + new URL("./gateway-tool.ts", import.meta.url), + "utf8", + ); + const m = /error: "env_value_is_placeholder",[\s\S]*?hint:\s*([\s\S]*?),\n\s*\};/.exec(src); + expect(m, "the placeholder branch must exist").not.toBeNull(); + return m![1]!; + } + + it("does NOT tell the caller to re-read the user's message for the value", () => { + expect(hint()).not.toMatch(/re-read the user's most recent message/i); + }); + + it("explicitly forbids asking the user to send the credential again", () => { + expect(hint()).toMatch(/DO NOT ask the user to send the credential again/i); + }); + + it("names the approval/redaction deadlock as the cause", () => { + const h = hint(); + expect(h).toMatch(/_confirmed: true/); + expect(h).toMatch(/redaction pass/i); + }); + + it("gives an out-of-band recovery that never routes the secret through a session", () => { + const h = hint(); + expect(h).toMatch(/comis secrets set/); + expect(h).toMatch(/never enters the session|encrypted store/i); + }); +}); diff --git a/packages/skills/src/platform-tools/tools/gateway-tool.ts b/packages/skills/src/platform-tools/tools/gateway-tool.ts index 3dd71f227a..f7f829dab8 100644 --- a/packages/skills/src/platform-tools/tools/gateway-tool.ts +++ b/packages/skills/src/platform-tools/tools/gateway-tool.ts @@ -348,10 +348,36 @@ export function createGatewayTool( if (envValue === "[REDACTED]" || /^\[REDACTED[^\]]*\]$/.test(envValue)) { return { error: "env_value_is_placeholder", + // The old hint said "re-read the user's most recent message and + // call env_set again with the literal value". That instruction is + // IMPOSSIBLE to follow after the redaction pass has run, and the + // only way an agent can comply is to ask the user to re-send the + // credential — which is exactly what happened live: a user was + // asked to paste their password into the chat a second time. + // + // Why it is impossible: an approval-gated env_set needs a SECOND + // call carrying `_confirmed: true`, but between the two calls the + // LCD/session redaction rewrites the secret-bearing fields of the + // user's own inbound message. `scrubRedactedToolCalls` removes the + // prior tool-call pair, so the agent has neither its own earlier + // argument nor the user's original text — the value is genuinely + // gone from context by the time consent arrives. + // + // So the hint must (a) name the deadlock, (b) forbid asking for + // re-transmission, and (c) give the path that never routes a + // credential through chat history at all. hint: - `env_value "${envValue}" is a session-redaction placeholder, ` + - `not a real secret. Re-read the user's most recent message ` + - `and call env_set again with the literal value they provided.`, + "env_value is a session-redaction placeholder, not a real secret. This is the " + + "approval/redaction deadlock, NOT a value the caller can recover: an " + + "approval-gated env_set requires a second call with `_confirmed: true`, and the " + + "redaction pass rewrites the secret in the originating message before that " + + "second call happens. " + + "DO NOT ask the user to send the credential again — re-transmission writes it " + + "into the conversation a second time and does not fix the gate. " + + "Tell the operator to set it outside the conversation instead: " + + "`comis secrets set ` on the host (it is written straight to the " + + "encrypted store and never enters the session), then retry the action that " + + "needed it.", }; } diff --git a/packages/skills/src/skills/integrations/mcp-client.test.ts b/packages/skills/src/skills/integrations/mcp-client.test.ts index f751526b7e..515b700a4c 100644 --- a/packages/skills/src/skills/integrations/mcp-client.test.ts +++ b/packages/skills/src/skills/integrations/mcp-client.test.ts @@ -9,6 +9,7 @@ import { describe, it, expect, vi, beforeEach, afterEach } from "vitest"; import type { TypedEventBus } from "@comis/core"; +import { MCP_CALL_TOOL_TIMEOUT_MS_DEFAULT } from "@comis/core"; // --------------------------------------------------------------------------- // Mock MCP SDK modules before importing the module under test @@ -604,10 +605,13 @@ describe("McpClientManager", () => { expect(result.value.content[0].text).toBe("result text"); expect(result.value.isError).toBe(false); + // The construction-time fallback is the SCHEMA default — asserting a local + // literal here is what let the client (60_000) drift from the schema + // (120_000), silently halving the deadline on any un-threaded wiring path. expect(mockCallTool).toHaveBeenCalledWith( { name: "search", arguments: { query: "test" } }, undefined, - { timeout: 60_000 }, + { timeout: MCP_CALL_TOOL_TIMEOUT_MS_DEFAULT }, ); }); diff --git a/packages/skills/src/skills/integrations/mcp-client/index.ts b/packages/skills/src/skills/integrations/mcp-client/index.ts index 097c2ad1e8..95eb4e3d69 100644 --- a/packages/skills/src/skills/integrations/mcp-client/index.ts +++ b/packages/skills/src/skills/integrations/mcp-client/index.ts @@ -17,6 +17,7 @@ */ import type { SystemIntervalHandle, SystemTimeoutHandle } from "@comis/core"; +import { MCP_CALL_TOOL_TIMEOUT_MS_DEFAULT } from "@comis/core"; import type PQueue from "p-queue"; import type { CircuitState, @@ -171,7 +172,10 @@ export function createMcpClientManager(deps: McpClientManagerDeps): McpClientMan // Resolve defaults at construction const options: McpClientManagerOptions = { connectTimeoutMs: deps.connectTimeoutMs ?? 30_000, - callToolTimeoutMs: deps.callToolTimeoutMs ?? 60_000, + // The schema's default is the ONE source of truth — a local literal here + // silently halved the deadline (60s) whenever the wiring did not thread the + // configured value through. See MCP_CALL_TOOL_TIMEOUT_MS_DEFAULT. + callToolTimeoutMs: deps.callToolTimeoutMs ?? MCP_CALL_TOOL_TIMEOUT_MS_DEFAULT, stdioDefaultConcurrency: deps.stdioDefaultConcurrency ?? 1, httpDefaultConcurrency: deps.httpDefaultConcurrency ?? 4, reconnectOpts: { diff --git a/packages/skills/src/skills/integrations/mcp-client/mcp-client-call.test.ts b/packages/skills/src/skills/integrations/mcp-client/mcp-client-call.test.ts index e695f9b681..1a268ab9ee 100644 --- a/packages/skills/src/skills/integrations/mcp-client/mcp-client-call.test.ts +++ b/packages/skills/src/skills/integrations/mcp-client/mcp-client-call.test.ts @@ -15,6 +15,7 @@ import { describe, it, expect, vi, beforeEach } from "vitest"; import PQueue from "p-queue"; import { UnauthorizedError } from "@modelcontextprotocol/sdk/client/auth.js"; +import { McpError, ErrorCode } from "@modelcontextprotocol/sdk/types.js"; import { runWithContext, type RequestContext } from "@comis/core"; import type { McpClientManagerDeps, @@ -171,6 +172,72 @@ describe("R8 needs_reauth", () => { }); }); +describe("call timeout names its knob", () => { + // Observed live: a heavy report tool hit the 120s + // `integrations.mcp.callToolTimeoutMs` FOUR times. The surfaced error was the bare + // SDK string "MCP error -32001: Request timed out", with a wrapper hint saying + // "retry the underlying operation when appropriate" — so the agent retried 4x, + // burned 8 minutes of the user's time, and tripped the circuit breaker. The error + // must name the knob + the value that expired, and must NOT invite a blind retry. + let logger: ReturnType; + let deps: McpClientManagerDeps; + + beforeEach(() => { + vi.clearAllMocks(); + logger = makeLogger(); + deps = { logger } as unknown as McpClientManagerDeps; + }); + + it("names integrations.mcp.callToolTimeoutMs and the configured value in the error", async () => { + const serverName = "vendor-mcp"; + const state = makeConnectedState(serverName, () => + Promise.reject(new McpError(ErrorCode.RequestTimeout, "Request timed out")), + ); + + const result = await callTool(state, deps, `mcp:${serverName}/vendor_activity_report`, {}); + + expect(result.ok).toBe(false); + if (result.ok) throw new Error("expected err"); + const message = result.error.message; + expect(message).toContain("integrations.mcp.callToolTimeoutMs"); + // the ACTUAL configured value, so the operator does not have to go look it up + expect(message).toContain(String(state.options.callToolTimeoutMs)); + expect(message).toContain(serverName); + }); + + it("tells the caller NOT to retry unchanged (the retry storm is the failure mode)", async () => { + const serverName = "vendor-mcp"; + const state = makeConnectedState(serverName, () => + Promise.reject(new McpError(ErrorCode.RequestTimeout, "Request timed out")), + ); + + const result = await callTool(state, deps, `mcp:${serverName}/vendor_activity_report`, {}); + if (result.ok) throw new Error("expected err"); + // An identical retry deterministically re-expires the same deadline. + expect(result.error.message).toMatch(/do not retry (it |the call )?unchanged/i); + // …and it names the two things that DO change the outcome. + expect(result.error.message).toMatch(/narrow/i); + }); + + it("emits a WARN carrying the same hint + a dependency errorKind", async () => { + const serverName = "vendor-mcp"; + const state = makeConnectedState(serverName, () => + Promise.reject(new McpError(ErrorCode.RequestTimeout, "Request timed out")), + ); + + await callTool(state, deps, `mcp:${serverName}/vendor_activity_report`, {}); + + const timeoutWarn = logger.warn.mock.calls.find( + (c) => typeof c[1] === "string" && /timed out/i.test(c[1] as string), + ); + expect(timeoutWarn).toBeDefined(); + const fields = timeoutWarn![0] as Record; + expect(fields.errorKind).toBe("dependency"); + expect(String(fields.hint)).toContain("integrations.mcp.callToolTimeoutMs"); + expect(fields.timeoutMs).toBe(state.options.callToolTimeoutMs); + }); +}); + describe("request correlation", () => { it("registers MCP progress handling while forwarding the request trace separately", async () => { const serverName = "inventory"; diff --git a/packages/skills/src/skills/integrations/mcp-client/mcp-client-call.ts b/packages/skills/src/skills/integrations/mcp-client/mcp-client-call.ts index bdf65cb1e4..08e83d2fe3 100644 --- a/packages/skills/src/skills/integrations/mcp-client/mcp-client-call.ts +++ b/packages/skills/src/skills/integrations/mcp-client/mcp-client-call.ts @@ -323,6 +323,7 @@ export async function callTool( logger.debug?.({ serverName, toolName }, "Tool call timed out, connection status preserved"); } + // Increment breaker on non-session-expired failures (includes timeouts + // post-call generation mismatches). isSessionExpired is EXEMPT above -- // it routes through handleDisconnection and the reconnect-success block @@ -350,7 +351,61 @@ export async function callTool( state.circuitBreakers.set(serverName, { status: cur.status, failureCount: newCount }); } + // A deadline expiry is DETERMINISTIC for the same call: the bare SDK string + // ("MCP error -32001: Request timed out") names neither the knob nor the + // value, so an agent reads it as transient and retries unchanged. Live + // (comis-moshe 2026-07-26): a month-wide 165-vehicle report re-expired the + // same 120s deadline FOUR times — 8 minutes of the user's time and a tripped + // breaker — because the surfaced hint said "retry the underlying operation + // when appropriate". Name the knob + the value that expired, and say plainly + // that an unchanged retry re-expires it. (AGENTS.md §2.7: a hint names the + // exact config key and the numbers that conflicted.) + // + // Placed AFTER the breaker accounting above so a timeout still counts toward + // the threshold exactly as before — only the surfaced message changes. + if (isTimeout) { + const timeoutMs = state.options.callToolTimeoutMs; + const timeoutHint = mcpCallTimeoutHint(serverName, toolName, timeoutMs); + logger.warn( + { + serverName, + toolName, + timeoutMs, + hint: timeoutHint, + errorKind: "dependency" as const, + }, + "MCP tool call timed out at the configured call deadline", + ); + return err(new Error(timeoutHint)); + } + return err(error instanceof Error ? error : new Error(message)); } }) as Promise>; } + +/** + * The operator/agent-facing message for an MCP call that hit its configured + * deadline. + * + * Exported so the log site, the returned `Error`, and the test all read the SAME + * text — a duplicated literal is how "No action needed."-class hints drift. + * + * @param serverName - the MCP server whose call expired. + * @param toolName - the qualified tool name that expired. + * @param timeoutMs - the ACTUAL resolved `integrations.mcp.callToolTimeoutMs`. + * @returns the hint text (also used verbatim as the Error message). + */ +export function mcpCallTimeoutHint( + serverName: string, + toolName: string, + timeoutMs: number, +): string { + return ( + `MCP tool "${toolName}" on server "${serverName}" timed out — it exceeded the call ` + + `deadline of ${timeoutMs}ms (\`integrations.mcp.callToolTimeoutMs\`, currently ${timeoutMs}). ` + + "This deadline is deterministic — do not retry it unchanged, the same call re-expires it. " + + "Either narrow the request (a smaller page/date window/fewer entities) so it completes " + + `inside ${timeoutMs}ms, or raise \`integrations.mcp.callToolTimeoutMs\` for this deployment.` + ); +} diff --git a/packages/skills/src/skills/integrations/mcp-client/mcp-client-types.ts b/packages/skills/src/skills/integrations/mcp-client/mcp-client-types.ts index 687571410e..7634c61894 100644 --- a/packages/skills/src/skills/integrations/mcp-client/mcp-client-types.ts +++ b/packages/skills/src/skills/integrations/mcp-client/mcp-client-types.ts @@ -317,7 +317,7 @@ export interface McpClientManagerDeps { readonly healthCheckIntervalMs?: number; /** Timeout for connect + listTools in milliseconds (default: 30000). */ readonly connectTimeoutMs?: number; - /** Timeout for individual callTool invocations in milliseconds (default: 60000). */ + /** Timeout for individual callTool invocations in milliseconds (default: {@link MCP_CALL_TOOL_TIMEOUT_MS_DEFAULT} = 120000). */ readonly callToolTimeoutMs?: number; /** Optional EventBus for emitting connection lifecycle events. */ readonly eventBus?: TypedEventBus; diff --git a/packages/web/src/api/contracts.generated.json b/packages/web/src/api/contracts.generated.json index b96c641732..f92d00035b 100644 --- a/packages/web/src/api/contracts.generated.json +++ b/packages/web/src/api/contracts.generated.json @@ -10990,6 +10990,10 @@ "totalTokens": { "type": "number" }, + "tokenBasis": { + "type": "string", + "const": "input+output+cache" + }, "cacheReadRatio": { "type": "number" } @@ -11391,6 +11395,12 @@ "downshifted", "exhausted" ] + }, + "freshTailSteps": { + "type": "number" + }, + "freshTailStepsConfigured": { + "type": "number" } }, "required": [ @@ -12460,6 +12470,12 @@ }, "deliveredWithToolErrors": { "type": "number" + }, + "hardDegraded": { + "type": "number" + }, + "hardDegradedRate": { + "type": "number" } }, "required": [ @@ -12531,6 +12547,10 @@ "totalTokens": { "type": "number" }, + "tokenBasis": { + "type": "string", + "const": "input+output" + }, "offSessionUsd": { "type": "number" } diff --git a/packages/web/src/api/contracts.generated.size.json b/packages/web/src/api/contracts.generated.size.json index 9a1c809a35..10d0d8e62e 100644 --- a/packages/web/src/api/contracts.generated.size.json +++ b/packages/web/src/api/contracts.generated.size.json @@ -1,6 +1,6 @@ { - "totalMinified": 186365, - "totalGzipped": 18652, + "totalMinified": 186601, + "totalGzipped": 18711, "budget": { "minified": 187000, "gzipped": 38912 @@ -157,12 +157,12 @@ "obs.delivery.recent": 2223, "obs.delivery.stats": 629, "obs.diagnostics": 596, - "obs.explain": 14368, + "obs.explain": 14508, "obs.getCacheStats": 358, "obs.reset": 571, "obs.reset.table": 483, "obs.spend.snapshot": 350, - "obs.system.health": 5183, + "obs.system.health": 5307, "obs.systemPromptReport.latest": 520, "obs.systemPromptReport.list": 503, "obs.trace.export": 362, diff --git a/packages/web/src/api/contracts.generated.ts b/packages/web/src/api/contracts.generated.ts index 53b67e36b3..096227b927 100644 --- a/packages/web/src/api/contracts.generated.ts +++ b/packages/web/src/api/contracts.generated.ts @@ -11166,6 +11166,10 @@ export const CONTRACTS = { "totalTokens": { "type": "number" }, + "tokenBasis": { + "type": "string", + "const": "input+output+cache" + }, "cacheReadRatio": { "type": "number" } @@ -11567,6 +11571,12 @@ export const CONTRACTS = { "downshifted", "exhausted" ] + }, + "freshTailSteps": { + "type": "number" + }, + "freshTailStepsConfigured": { + "type": "number" } }, "required": [ @@ -12636,6 +12646,12 @@ export const CONTRACTS = { }, "deliveredWithToolErrors": { "type": "number" + }, + "hardDegraded": { + "type": "number" + }, + "hardDegradedRate": { + "type": "number" } }, "required": [ @@ -12707,6 +12723,10 @@ export const CONTRACTS = { "totalTokens": { "type": "number" }, + "tokenBasis": { + "type": "string", + "const": "input+output" + }, "offSessionUsd": { "type": "number" } diff --git a/test/architecture/daemon-boot-degrades-without-autonomy.test.ts b/test/architecture/daemon-boot-degrades-without-autonomy.test.ts new file mode 100644 index 0000000000..3955977202 --- /dev/null +++ b/test/architecture/daemon-boot-degrades-without-autonomy.test.ts @@ -0,0 +1,81 @@ +// SPDX-License-Identifier: Apache-2.0 +/** + * Boot-path guard: a SUPPORTED config must never crash-loop the daemon. + * + * The defect this pins (found by standing up a live rig, 2026-07-26): + * `constructCapabilityLayer` returns `capEndpointHandle: undefined` BY DESIGN + * when no agent has autonomy enabled. `setupProactiveSchedulers` treats that + * handle as mandatory, and `daemon.ts` threw on the failed Result — so the daemon + * completed its ENTIRE boot (channels registered, adapter polling) and then + * exited 1, forever. `systemctl is-active` reported `active` the whole time while + * the box served nothing; 13 restarts in five minutes. + * + * Reachability is the severity: an omitted `autonomy` block defaults to ENABLED, + * but writing any sub-key without `enabled: true` — e.g. the documented + * `autonomy.durability` — resolves to DISABLED. So adding one documented knob + * bricked the daemon. + * + * Composition-root wiring is not reachable from a handler unit test (the handler + * stays green while the live boot dies), so this is a source guard — the shape + * `02-DISCIPLINE.md` prescribes for built-but-not-wired defects. + */ +import { describe, it, expect } from "vitest"; +import * as fs from "node:fs"; +import * as path from "node:path"; +import { fileURLToPath } from "node:url"; +import { resolveAutonomy } from "@comis/core"; + +const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "../.."); +const daemonSrc = fs.readFileSync(path.join(repoRoot, "packages/daemon/src/daemon.ts"), "utf8"); + +describe("autonomy-disabled is a reachable, supported config", () => { + it("an omitted autonomy block defaults to ENABLED", () => { + expect(resolveAutonomy(undefined).enabled).toBe(true); + }); + + it("an explicit disable resolves DISABLED", () => { + expect(resolveAutonomy({ enabled: false } as never).enabled).toBe(false); + }); + + it("a sub-key WITHOUT `enabled: true` also resolves DISABLED (the reachability trap)", () => { + const durabilityOnly = { + durability: { enabled: true, orchestrateResume: true }, + profile: "assistant", + } as never; + expect(resolveAutonomy(durabilityOnly).enabled).toBe(false); + }); +}); + +describe("the daemon boots when proactive schedulers cannot be armed", () => { + it("does NOT throw unconditionally on a failed proactive-scheduler Result", () => { + // The regression shape: `if (!proactive.ok) { throw ... }` with no branch. + const unconditionalThrow = + /if \(!proactive\.ok\) \{\s*throw new Error\(`Proactive scheduler activation failed/; + expect(daemonSrc).not.toMatch(unconditionalThrow); + }); + + it("distinguishes the SUPPORTED missing-capEndpoint case from a real wiring regression", () => { + expect(daemonSrc).toContain("assertProactiveFailureIsSupported"); + // Any other missing dependency must still abort — a composition-root + // regression is NOT something to boot through. + const degrade = fs.readFileSync(path.join(repoRoot, "packages/daemon/src/wiring/proactive-degrade.ts"), "utf8"); + expect(degrade).toMatch(/isAutonomyDisabledProactiveMiss\(proactive\.error\)[\s\S]{0,120}throw new Error/); + }); + + it("logs an ERROR naming what is off and the knob that turns it back on", () => { + const degradeSrc = fs.readFileSync( + path.join(repoRoot, "packages/daemon/src/wiring/proactive-degrade.ts"), + "utf8", + ); + expect(degradeSrc).toContain("Proactive schedulers not armed"); + expect(degradeSrc).toContain("agents..autonomy.enabled: true"); + // …and warns about the sub-key trap that makes this reachable by accident. + expect(degradeSrc).toMatch(/autonomy\.durability/); + // the daemon must actually EMIT it + expect(daemonSrc).toContain("proactiveNotArmedLogFields()"); + }); + + it("treats the proactive surface as optional downstream (no undefined deref)", () => { + expect(daemonSrc).toMatch(/\.\.\.\(proactive\.ok \? \{ proactiveSchedulers: proactive\.value \} : \{\}\)/); + }); +}); diff --git a/test/architecture/mcp-timeout-default-parity.test.ts b/test/architecture/mcp-timeout-default-parity.test.ts new file mode 100644 index 0000000000..6b11fc9adc --- /dev/null +++ b/test/architecture/mcp-timeout-default-parity.test.ts @@ -0,0 +1,59 @@ +// SPDX-License-Identifier: Apache-2.0 +/** + * Parity gate: the MCP call-tool deadline has ONE default. + * + * The two-sources-of-truth drift this pins (comis-moshe 2026-07-26): the config + * schema defaulted `integrations.mcp.callToolTimeoutMs` to 120_000 while + * `createMcpClientManager` fell back to a local `?? 60_000` literal and the + * `McpClientManagerDeps` JSDoc documented 60000. Any wiring path that did not + * thread the configured value therefore ran a silently-halved deadline, and the + * docs told an operator the wrong number. + * + * Source-derived on purpose: reading the literals out of the files is what + * catches a future edit to either side. + */ +import { describe, it, expect } from "vitest"; +import * as fs from "node:fs"; +import * as path from "node:path"; +import { fileURLToPath } from "node:url"; +import { McpConfigSchema, MCP_CALL_TOOL_TIMEOUT_MS_DEFAULT } from "@comis/core"; + +const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "../.."); + +describe("MCP callToolTimeoutMs has a single default", () => { + it("the schema's parsed default IS the exported constant", () => { + const parsed = McpConfigSchema.parse({}); + expect(parsed.callToolTimeoutMs).toBe(MCP_CALL_TOOL_TIMEOUT_MS_DEFAULT); + }); + + it("the client manager's construction fallback references the constant, never a literal", () => { + const src = fs.readFileSync( + path.join(repoRoot, "packages/skills/src/skills/integrations/mcp-client/index.ts"), + "utf8", + ); + const line = src + .split("\n") + .find((l) => l.includes("callToolTimeoutMs:") && l.includes("??")); + expect(line, "no callToolTimeoutMs fallback line found").toBeDefined(); + expect(line).toContain("MCP_CALL_TOOL_TIMEOUT_MS_DEFAULT"); + // A bare numeric fallback is exactly the drift this gate exists to stop. + expect(line).not.toMatch(/\?\?\s*\d/); + }); + + it("the deps JSDoc does not document a stale numeric default", () => { + const src = fs.readFileSync( + path.join(repoRoot, "packages/skills/src/skills/integrations/mcp-client/mcp-client-types.ts"), + "utf8", + ); + const jsdoc = src + .split("\n") + .filter((l) => /callToolTimeoutMs invocations|individual callTool invocations/.test(l)); + expect(jsdoc.length).toBeGreaterThan(0); + for (const line of jsdoc) { + const numbers = line.match(/\d{4,}/g) ?? []; + for (const n of numbers) { + expect(Number(n)).toBe(MCP_CALL_TOOL_TIMEOUT_MS_DEFAULT); + } + } + }); +}); diff --git a/test/architecture/token-basis-lens-reconciliation.test.ts b/test/architecture/token-basis-lens-reconciliation.test.ts new file mode 100644 index 0000000000..5a55279225 --- /dev/null +++ b/test/architecture/token-basis-lens-reconciliation.test.ts @@ -0,0 +1,48 @@ +// SPDX-License-Identifier: Apache-2.0 +/** + * Cross-lens gate: `totalTokens` must declare which convention it counts. + * + * The drift this pins: `IncidentReport.cost.totalTokens` counts input + output + + * CACHE, while `SystemHealthReport.cost.totalTokens` counts input + output only. + * One 27-minute session legitimately reported 6,043,245 on one lens and 18,637 on + * the other, and neither JSON said which convention it followed — so a reader + * comparing them concludes a lens is broken. The `tokenBasis` discriminator makes + * the two reconcilable programmatically. + */ +import { describe, it, expect } from "vitest"; +import * as fs from "node:fs"; +import * as path from "node:path"; +import { fileURLToPath } from "node:url"; + +const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "../.."); + +function read(rel: string): string { + return fs.readFileSync(path.join(repoRoot, rel), "utf8"); +} + +describe("totalTokens declares its counting convention on every lens", () => { + it("the IncidentReport schema declares the cache-inclusive basis", () => { + const src = read("packages/core/src/api-contracts/incident-report.ts"); + expect(src).toContain('tokenBasis: z.literal("input+output+cache")'); + }); + + it("the SystemHealthReport schema declares the cache-exclusive basis", () => { + const src = read("packages/core/src/api-contracts/system-health-report.ts"); + expect(src).toContain('tokenBasis: z.literal("input+output")'); + }); + + it("the two lenses declare DIFFERENT bases (the whole point of the field)", () => { + const incident = read("packages/core/src/api-contracts/incident-report.ts"); + const system = read("packages/core/src/api-contracts/system-health-report.ts"); + expect(incident).toContain("input+output+cache"); + // …and the system lens must NOT claim the cache-inclusive basis. + expect(system).not.toContain('z.literal("input+output+cache")'); + }); + + it("each assembler STAMPS its basis (a schema field nothing populates is dead)", () => { + expect(read("packages/daemon/src/api/obs-handlers/obs-explain-assemble.ts")) + .toContain('tokenBasis: "input+output+cache"'); + expect(read("packages/daemon/src/api/obs-handlers/system-health.ts")) + .toContain('tokenBasis: "input+output"'); + }); +}); From 4fb6e66696c5cb0cb09d9e2cd9a2ed6148c8ee91 Mon Sep 17 00:00:00 2001 From: comis-agent Date: Sun, 26 Jul 2026 19:35:47 +0300 Subject: [PATCH 02/33] docs(live-test): record two rig gotchas that read as a failed deploy MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both cost a cycle while standing up a live-test box. A HEAD-only symbol grep run as root returns a FALSE NEGATIVE: /home/comis is 0700, so the Permission-denied goes to stderr while `grep -rl ... | wc -l` prints 0 on stdout. That is indistinguishable from "the deploy did not land" and sends you chasing a packaging bug that does not exist. Run the proof as the service user, and cross-check the packed tarball when in doubt. `sudo -u comis -i bash -lc "..."` mangles a multi-line loop into `syntax error near unexpected token 'do'` — use a `bash -s` heredoc instead. Same trap family as the existing `su - comis -c` quoting note. --- test/live/self-driving/01-SETUP.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/test/live/self-driving/01-SETUP.md b/test/live/self-driving/01-SETUP.md index 742906261b..6f18c52de8 100644 --- a/test/live/self-driving/01-SETUP.md +++ b/test/live/self-driving/01-SETUP.md @@ -169,6 +169,16 @@ Do not start the real test plan until ALL hold: - **`pkill -f "daemon.js"` self-matches your ssh shell** (its argv contains the pattern) → kills the shell → ssh exit 255. Always anchor: `pkill -9 -f "^node .*daemon\.js"`. **Same trap for the emulator:** `pkill -f "vps-emu"` self-kills the ssh shell running it (argv has "vps-emu.ts") — anchor `pkill -9 -f "^node .*vps-emu"`. And a backgrounded `nohup/setsid … &` emulator dies on ssh close → launch it in **tmux**. Both are baked into **`scripts/restart-emu.sh`** (use it; the port is kernel-allocated, so re-wire after: `node /root/wire-emu.mjs && bash /root/restart-daemon.sh`). - **Severing the LCD needs the FORMATTED session key, not the trajectory-filename form.** `session.reset_conversation {session_key}` wants `default:::peer:` (read it from `db.mjs sql "SELECT DISTINCT session_key FROM lcd_messages"`), NOT the `~`-separated trajectory filename (`~peer~`). On a key-format mismatch it returns `lcdRowsDeleted:0` **silently** (no error) → the LCD is NOT cleared and a "cross-session" recall test is invalid. Verify `lcdRowsDeleted>0` after a sever. - **Media OUTPUT delivery (image-gen/TTS/video-gen) is observable on the channel oracle** — the emulator records `sendPhoto`/`sendAudio`/`sendVideo` (with `mediaKind`), not just `sendVoice`/`sendDocument`. A media-only turn delivers no text, so `drive.mjs` prints `[NO SUBSTANTIVE ANSWER]` — read the outbound (`…/outbound` shows the `sendAudio`/`sendPhoto`) not the drive's text verdict. +- **A symbol grep run as ROOT returns a FALSE NEGATIVE — `/home/comis` is `0700`.** The + HEAD-only symbol proof (§2b) must run as the service user: + `ssh $VPS 'sudo -u comis -i bash -s' <<'EOS' … EOS`. Run as root it prints + `Permission denied` on stderr and `grep -rl … | wc -l` reports **0 files** on stdout — + which reads exactly like "the deploy didn't land / the build is stale" and sends you + chasing a non-existent packaging bug. Verify the tarball too when in doubt: + `tar xzOf packages/comis/comisai-*.tgz 'package/node_modules/@comis//dist/.js' | grep -c `. +- **`sudo -u comis -i bash -lc "…"` mangles multi-line loops** (the newlines collapse and + you get `syntax error near unexpected token 'do'`). Use a heredoc into + `bash -s` instead — same trap family as the `su - comis -c` quoting note below. - **SSH drops on long sleeps** → add `-o ServerAliveInterval=5`. macOS has no `timeout`. - **`pgrep -f daemon.js` false-matches** your `bash -lc`/`sudo` wrappers → filter `grep -vE "bash|sudo|grep"`. - **Don't chase an `effectiveWindow:8192` viable-floor WARN** — for a catalog-unknown model it's usually an invalid/mistyped model, not a window bug. Pick a valid model. From 750b9d61022ab617be10ca3ab970f384423d062f Mon Sep 17 00:00:00 2001 From: comis-agent Date: Sun, 26 Jul 2026 20:42:15 +0300 Subject: [PATCH 03/33] fix(activity,security,obs): close every remaining finding from the incident run MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Second pass over the same live incident: the four ledgered findings plus the small frictions, each fixed test-first and the headline one re-proven on the wire. Backgrounded tools tell the truth end to end. A background hand-off returns a non-error placeholder, so the activity card closed a still-running tool as "completed" — four reports that later timed out were each shown as succeeded, and the turn then finalized failure with nothing to name, pinning a failure pill above a delivered answer. Four coupled layers: * tool:executed carries `backgrounded`; the activity stream leaves the card running on a hand-off instead of closing it. * background_task:{completed,failed} carry the promote-time toolCallId/ sessionKey/traceId (captured in the middleware, persisted on the task), so the terminal event can close the card it belongs to — previously impossible: the events had only taskId while cards are keyed on toolCallId. * The stream closes the card on the real terminal (redeliveries skipped; a pre-upgrade record without correlation is a no-op). * A failure whose answer WAS fully delivered reclassifies to success_with_recovered_failures when the failure is attributable to observed events — never plain success (no events ⇒ the truthful failure stands), and never for a resource abort (a stopped run renders as stopped). Live re-proof on the rig, same turn shape as the incident: in-flight failures shown WITH the tool name, artifact delivered, card ends "✓ done" and is removed, finalize reads success_with_recovered_failures with failedEventCount 2 — where the incident showed a kept "❌ dependency" over the user's file. Secret field names match on keyword BOUNDARIES, pairs-only. The suffix-anchored pattern missed any credential keyword followed by more name (AWS_BEARER_TOKEN_BEDROCK, SECRETS_MASTER_KEY). The fix matches qualifier+keyword segment pairs (bearer+token, master+key, client+secret, db+password, …) rather than bare keywords: a first cut with bare `secret` false-flagged the stock `security.writeSecretGuard` boolean, which made persistToConfig's plaintext-secret guard abort — every config write would have failed. The default config now scans clean by construction, and meta names (secretName, passwordResetRequired) are pinned negative alongside the counting vocabulary. A gated secret write survives its own confirmation. The approval gate needs a second call, and session redaction rewrites the secret in context between the two — the only recovery was asking the user to re-paste a credential into the chat. env_set now stashes the request server-side and returns a pending_action_id; the confirm passes the id back and the daemon replays the original value (one-shot, 5-minute TTL), even when the model's context re-supplies the redaction placeholder. The secret never has to survive in conversation history. Audit rows are joinable. secret:accessed carries optional sessionKey/traceId; the sink prefers them over AsyncLocalStorage (empty for boot reads — why every live row had traceId null) and tags refs.origin boot|request so no-context-by-design is distinguishable from context-lost. PATH-class environment names no longer dilute the trail. Also: the LLM-input debug line carries a digest + executionId so it joins to the trajectory's prompt.submitted record; URL extraction strips the quote family (a config snippet's closing quote rode into the fetch and 404'd); a PEP 668 pip refusal gets a recovery hint naming the venv and --break-system-packages paths up front; the learning_health finding points at the outcome-judge no-op when coverage starves; the live-test env template uses default-assign so an inline override can no longer be clobbered into targeting the wrong box. --- docs/agent-tools/infrastructure.mdx | 14 +- .../auto-background-middleware.test.ts | 4 + .../background/auto-background-middleware.ts | 12 +- .../background-task-manager.test.ts | 47 +++++++ .../src/background/background-task-manager.ts | 23 +++- .../src/background/background-task-types.ts | 7 + packages/agent/src/bridge/pi-event-bridge.ts | 11 ++ .../src/executor/prompt-runner/retry-loop.ts | 17 ++- packages/core/src/activity/turn-outcome.ts | 14 +- packages/core/src/event-bus/events-agent.ts | 10 ++ packages/core/src/event-bus/events-infra.ts | 29 ++++ packages/core/src/exports/activity.ts | 1 + .../src/security/secret-detection.test.ts | 77 ++++++----- .../core/src/security/secret-detection.ts | 89 ++++++++++++- .../src/api/obs-handlers/system-findings.ts | 2 +- .../src/observability/obs-audit-sink.test.ts | 67 ++++++++++ .../src/observability/obs-audit-sink.ts | 31 ++++- .../src/activity/activity-stream.test.ts | 125 +++++++++++++++++ .../src/activity/activity-stream.ts | 48 +++++++ .../activity-turn-coordinator.test.ts | 73 ++++++++++ .../execution/activity-turn-coordinator.ts | 19 +++ .../src/execution/execution-pipeline.ts | 16 ++- .../src/execution/turn-outcome-mapper.test.ts | 43 +++++- .../src/execution/turn-outcome-mapper.ts | 29 +++- .../tool-registry.typebox-1.3.6.snap | 4 + .../platform-tools/tools/gateway-tool.test.ts | 126 ++++++++++++++++-- .../src/platform-tools/tools/gateway-tool.ts | 123 ++++++++++++++++- .../tools/builtin/exec-diagnostics.test.ts | 24 ++++ .../src/tools/builtin/exec-diagnostics.ts | 21 +++ .../integrations/link/link-detector.test.ts | 15 +++ .../tools/integrations/link/link-detector.ts | 7 +- .../self-driving/scripts/.live-env.example | 30 +++-- 32 files changed, 1081 insertions(+), 77 deletions(-) create mode 100644 packages/daemon/src/observability/obs-audit-sink.test.ts diff --git a/docs/agent-tools/infrastructure.mdx b/docs/agent-tools/infrastructure.mdx index cc1731b06f..ef6c091fe8 100644 --- a/docs/agent-tools/infrastructure.mdx +++ b/docs/agent-tools/infrastructure.mdx @@ -119,11 +119,23 @@ The `gateway` tool is the Swiss Army knife for system management, with **11 acti |-----------|------|----------|-------------| | `action` | string | Yes | `"env_set"` | | `env_key` | string | Yes | Environment variable / secret name (uppercase, e.g., `OPENAI_API_KEY`) | - | `env_value` | string | Yes | Secret value to store. Write-only: cannot be read back. | + | `env_value` | string | On the first call | Secret value to store. Write-only: cannot be read back. Omit on the confirm leg (see below). | + | `pending_action_id` | string | On the confirm leg | The id returned by the gated first call. Pass it with `_confirmed: true` **instead of** re-sending `env_value`. | Setting environment variables requires security approval and user confirmation. The stored value is write-only and cannot be retrieved. Reference it in config as `${VAR_NAME}` (e.g., `GEMINI_API_KEY: ${GEMINI_API_KEY}`). Never write raw API keys into config files. + + **The two-call approval flow never re-sends the secret.** The gated first call + stashes the value server-side (5-minute TTL, one-shot) and returns a + `pending_action_id`; after the user approves, the confirm call passes + `{ _confirmed: true, pending_action_id }` and the daemon replays the stashed + value. This matters because session redaction rewrites the secret in + conversation context between the two calls — re-sending `env_value` on the + confirm leg risks transmitting a redaction placeholder (rejected), and asking + the user to re-paste a credential into the chat is never acceptable. An + expired or already-consumed id returns `pending_action_expired`; re-initiate, + or have the operator run `comis secrets set ` on the host. diff --git a/packages/agent/src/background/auto-background-middleware.test.ts b/packages/agent/src/background/auto-background-middleware.test.ts index 6c83fc4491..4588321e4c 100644 --- a/packages/agent/src/background/auto-background-middleware.test.ts +++ b/packages/agent/src/background/auto-background-middleware.test.ts @@ -417,6 +417,10 @@ describe("wrapToolForAutoBackground", () => { expect.any(Promise), expect.any(AbortController), expectedOrigin, + undefined, + // Ids-only correlation captured at promote time so the TERMINAL event + // can close the activity card this tool call opened. + expect.objectContaining({ toolCallId: "call-1" }), ); }); diff --git a/packages/agent/src/background/auto-background-middleware.ts b/packages/agent/src/background/auto-background-middleware.ts index 26b791507a..b5ff7d7ff4 100644 --- a/packages/agent/src/background/auto-background-middleware.ts +++ b/packages/agent/src/background/auto-background-middleware.ts @@ -9,6 +9,7 @@ * @module */ import { suppressError } from "@comis/shared"; +import { tryGetContext } from "@comis/core"; import type { BackgroundTasksConfig } from "@comis/core"; import { systemSetTimeout, systemClearTimeout } from "@comis/core"; import type { AgentToolResult } from "@earendil-works/pi-agent-core"; @@ -165,7 +166,16 @@ export function wrapToolForAutoBackground( return await taskPromise; } - const promoteResult = manager.promote(tool.name, taskPromise, ac, origin); + // Ids-only correlation so the TERMINAL background event can close the + // activity card this tool call opened (the card is keyed on the + // toolCallId; without it a backgrounded tool's lifecycle was closed + // "completed" at hand-off and the real outcome never reached the card). + const turnCtx = tryGetContext(); + const promoteResult = manager.promote(tool.name, taskPromise, ac, origin, undefined, { + toolCallId, + ...(turnCtx?.sessionKey !== undefined ? { sessionKey: turnCtx.sessionKey } : {}), + ...(turnCtx?.traceId !== undefined ? { traceId: turnCtx.traceId } : {}), + }); if (!promoteResult.ok) { // Concurrency limit hit: fall back to foreground (await normally) return await taskPromise; diff --git a/packages/agent/src/background/background-task-manager.test.ts b/packages/agent/src/background/background-task-manager.test.ts index 7d49c855a3..2930b3b42f 100644 --- a/packages/agent/src/background/background-task-manager.test.ts +++ b/packages/agent/src/background/background-task-manager.test.ts @@ -354,6 +354,53 @@ describe("BackgroundTaskManager", () => { }); }); + describe("terminal-event correlation (F-ACT-1 layer 2)", () => { + // The activity card keys a tool's lifecycle on tool:. Without + // this correlation the terminal event could not close the card it belongs + // to, so a backgrounded tool was closed "completed" at hand-off — the card + // told a live user four failed reports had succeeded. + const CORR = { + toolCallId: "call-9", + sessionKey: "default:agent:default:u:telegram:peer:u", + traceId: "trace-9", + }; + + it("promote stores the correlation and fail() emits it on the terminal event", () => { + const origin = buildOrigin({ agentId: "agent-1" }); + const r = manager.promote("report", new Promise(() => {}), new AbortController(), origin, undefined, CORR); + expect(r.ok).toBe(true); + if (!r.ok) return; + manager.fail(r.value, new Error("timed out")); + expect(eventBus.emit).toHaveBeenCalledWith( + "background_task:failed", + expect.objectContaining(CORR), + ); + }); + + it("complete() emits the same correlation", () => { + const origin = buildOrigin({ agentId: "agent-1" }); + const r = manager.promote("report", new Promise(() => {}), new AbortController(), origin, undefined, CORR); + if (!r.ok) throw new Error("promote failed"); + manager.complete(r.value, "done"); + expect(eventBus.emit).toHaveBeenCalledWith( + "background_task:completed", + expect.objectContaining(CORR), + ); + }); + + it("a promote WITHOUT correlation emits terminals without the fields (pre-upgrade shape)", () => { + const origin = buildOrigin({ agentId: "agent-1" }); + const r = manager.promote("report", new Promise(() => {}), new AbortController(), origin); + if (!r.ok) throw new Error("promote failed"); + manager.fail(r.value, new Error("boom")); + const call = (eventBus.emit as ReturnType).mock.calls.find( + (c) => c[0] === "background_task:failed", + ); + expect(call).toBeDefined(); + expect((call![1] as Record).toolCallId).toBeUndefined(); + }); + }); + describe("complete", () => { it("sets status completed with truncated result and decrements counters", () => { const result = manager.promote("tool", new Promise(() => {}), new AbortController(), buildOrigin({ agentId: "agent-1" })); diff --git a/packages/agent/src/background/background-task-manager.ts b/packages/agent/src/background/background-task-manager.ts index 1f7e74740f..b7d4adc3c4 100644 --- a/packages/agent/src/background/background-task-manager.ts +++ b/packages/agent/src/background/background-task-manager.ts @@ -71,6 +71,9 @@ export interface BackgroundTaskManager { ac: AbortController, origin: BackgroundTaskOrigin, notificationPolicy?: BackgroundTaskNotificationPolicy, + /** Ids-only correlation to the originating tool call + turn — lets the + * terminal event close the right activity card (see BackgroundTask). */ + correlation?: { toolCallId?: string; sessionKey?: string; traceId?: string }, ): Result; complete(taskId: string, result: unknown): Result; fail(taskId: string, error: unknown): Result; @@ -296,7 +299,7 @@ export function createBackgroundTaskManager(opts: BackgroundTaskManagerOpts): Ba } const manager: BackgroundTaskManager = { - promote(toolName, promise, ac, origin, notificationPolicy) { + promote(toolName, promise, ac, origin, notificationPolicy, correlation) { const parsedOrigin = BackgroundTaskOriginSchema.safeParse(origin); if (!parsedOrigin.success) { return err(new Error("BackgroundTaskOrigin requires valid structured turn authority")); @@ -320,6 +323,9 @@ export function createBackgroundTaskManager(opts: BackgroundTaskManagerOpts): Ba startedAt: clock.now(), origin: acceptedOrigin, notificationPolicy: notificationPolicy ?? "deferred", + ...(correlation?.toolCallId !== undefined ? { toolCallId: correlation.toolCallId } : {}), + ...(correlation?.sessionKey !== undefined ? { sessionKey: correlation.sessionKey } : {}), + ...(correlation?.traceId !== undefined ? { traceId: correlation.traceId } : {}), dispatchState: "pending", continuationExecutionId: taskId, dispatchAttempts: 0, @@ -407,6 +413,9 @@ export function createBackgroundTaskManager(opts: BackgroundTaskManagerOpts): Ba durationMs, origin: task.origin, timestamp: clock.now(), + ...(task.toolCallId !== undefined ? { toolCallId: task.toolCallId } : {}), + ...(task.sessionKey !== undefined ? { sessionKey: task.sessionKey } : {}), + ...(task.traceId !== undefined ? { traceId: task.traceId } : {}), }); return ok(undefined); }, @@ -435,6 +444,9 @@ export function createBackgroundTaskManager(opts: BackgroundTaskManagerOpts): Ba durationMs, origin: task.origin, timestamp: clock.now(), + ...(task.toolCallId !== undefined ? { toolCallId: task.toolCallId } : {}), + ...(task.sessionKey !== undefined ? { sessionKey: task.sessionKey } : {}), + ...(task.traceId !== undefined ? { traceId: task.traceId } : {}), }); return ok(undefined); }, @@ -587,6 +599,9 @@ export function createBackgroundTaskManager(opts: BackgroundTaskManagerOpts): Ba durationMs: (persisted.completedAt ?? clock.now()) - persisted.startedAt, origin: task.origin, timestamp: clock.now(), + ...(task.toolCallId !== undefined ? { toolCallId: task.toolCallId } : {}), + ...(task.sessionKey !== undefined ? { sessionKey: task.sessionKey } : {}), + ...(task.traceId !== undefined ? { traceId: task.traceId } : {}), }); } else if (persisted.status === "failed") { count++; @@ -598,6 +613,9 @@ export function createBackgroundTaskManager(opts: BackgroundTaskManagerOpts): Ba durationMs: (persisted.completedAt ?? clock.now()) - persisted.startedAt, origin: task.origin, timestamp: clock.now(), + ...(task.toolCallId !== undefined ? { toolCallId: task.toolCallId } : {}), + ...(task.sessionKey !== undefined ? { sessionKey: task.sessionKey } : {}), + ...(task.traceId !== undefined ? { traceId: task.traceId } : {}), }); } } @@ -778,6 +796,9 @@ export function createBackgroundTaskManager(opts: BackgroundTaskManagerOpts): Ba durationMs: (current.completedAt ?? clock.now()) - current.startedAt, origin: current.origin, timestamp: clock.now(), + ...(current.toolCallId !== undefined ? { toolCallId: current.toolCallId } : {}), + ...(current.sessionKey !== undefined ? { sessionKey: current.sessionKey } : {}), + ...(current.traceId !== undefined ? { traceId: current.traceId } : {}), // This is a DISPATCH redrive of an already-terminal task, not a new // terminal transition. Marked so occurrence-counting consumers (the // trajectory bridge) record the failure ONCE — the unmarked re-emit diff --git a/packages/agent/src/background/background-task-types.ts b/packages/agent/src/background/background-task-types.ts index 7278b265b1..179cf69baa 100644 --- a/packages/agent/src/background/background-task-types.ts +++ b/packages/agent/src/background/background-task-types.ts @@ -152,6 +152,13 @@ export interface BackgroundTask { /** Live notification policy. Optional; recovery defaults to "deferred" when * absent. */ notificationPolicy?: BackgroundTaskNotificationPolicy; + /** Correlation to the originating tool call + turn, captured at promote time + * and PERSISTED (ids only — no content), so the terminal + * `background_task:{completed,failed}` event can close the right activity + * card even across a daemon restart. Absent on pre-upgrade records. */ + toolCallId?: string; + sessionKey?: string; + traceId?: string; /** Durable completion lifecycle. */ dispatchState?: BackgroundSessionState; continuationExecutionId: string; diff --git a/packages/agent/src/bridge/pi-event-bridge.ts b/packages/agent/src/bridge/pi-event-bridge.ts index f1753ffe4a..9643739ef9 100644 --- a/packages/agent/src/bridge/pi-event-bridge.ts +++ b/packages/agent/src/bridge/pi-event-bridge.ts @@ -1376,11 +1376,22 @@ export function createPiEventBridge(deps: PiEventBridgeDeps): PiEventBridgeResul ? extractWebResultMetadata(endEvent.toolName, endEvent.result) : undefined; + // A background HAND-OFF is not an outcome: the middleware returns a + // non-error placeholder carrying details.status:"backgrounded", so + // toolSuccess is true while the tool is still running. Mark it so + // outcome consumers (the activity card above all) do not close a + // still-running tool as "completed". + const resultBackgrounded = + endEvent.result != null + && typeof endEvent.result === "object" + && ((endEvent.result as Record).details as Record | undefined) + ?.status === "backgrounded"; deps.eventBus.emit("tool:executed", { toolName: endEvent.toolName, toolCallId: endEvent.toolCallId, durationMs, success: toolSuccess, + ...(resultBackgrounded ? { backgrounded: true } : {}), timestamp: systemNowMs(), agentId: deps.agentId, sessionKey: formatSessionKey(deps.sessionKey), diff --git a/packages/agent/src/executor/prompt-runner/retry-loop.ts b/packages/agent/src/executor/prompt-runner/retry-loop.ts index 712233e982..7fa10ee7b2 100644 --- a/packages/agent/src/executor/prompt-runner/retry-loop.ts +++ b/packages/agent/src/executor/prompt-runner/retry-loop.ts @@ -1,4 +1,5 @@ // SPDX-License-Identifier: Apache-2.0 +import { createHash } from "node:crypto"; /** * Model retry orchestration — wraps `runWithModelRetry` and layers on * stuck-session detection plus the silent-failure detection cascade @@ -75,9 +76,19 @@ export async function runRetryLoop( promptError: undefined, }; - // Redact LLM input -- log only character count, never user - // message text, canary tokens, or system prompt content. - deps.logger.debug({ inputChars: messageText.length }, "LLM input"); + // Redact LLM input -- log only character count + a correlation digest, never + // user message text, canary tokens, or system prompt content. The digest + // joins this line to the trajectory's prompt.submitted record (which carries + // the same identity), and the VALUES live in the raw session .jsonl -- this + // line deliberately cannot reconstruct the prompt, only point at what can. + deps.logger.debug( + { + inputChars: messageText.length, + inputDigest: createHash("sha256").update(messageText).digest("hex").slice(0, 16), + executionId: params.executionId, + }, + "LLM input", + ); // Bind the model-retry invocation so the silent-failure branches share // the deps wiring without re-threading every dependency. diff --git a/packages/core/src/activity/turn-outcome.ts b/packages/core/src/activity/turn-outcome.ts index 7e2a7df865..ca8b8a1680 100644 --- a/packages/core/src/activity/turn-outcome.ts +++ b/packages/core/src/activity/turn-outcome.ts @@ -80,7 +80,19 @@ export type TurnOutcome = * rendered status reads truthfully instead of the bare errorKind. A * closed-vocabulary named-constant string — never raw provider/internal text. */ - reason?: string } + reason?: string; + /** + * Present when the final answer WAS fully delivered despite the execution + * error — the evidence the coordinator needs to reclassify this turn to + * `success_with_recovered_failures` (answer shown without a failure pill; + * the failure preserved on the outcome and `activity:turn_finalized`). + * Live shape: a backgrounded report timed out (execution lifecycle + * "error") while a later retry delivered the artifact — the user got + * their file with a "❌ dependency" pill above it. NEVER set for a + * resource abort (`reason` present): a stopped run must render as + * stopped even when partial text delivered. + */ + delivery?: FinalDeliveryReceipt } | { kind: "silent"; reason: "SILENT" | "HEARTBEAT_OK" | "NO_REPLY" } | { kind: "aborted"; reason: "user_cancel" | "timeout" | "fatal" }; diff --git a/packages/core/src/event-bus/events-agent.ts b/packages/core/src/event-bus/events-agent.ts index 10277dbeff..787cf6fb36 100644 --- a/packages/core/src/event-bus/events-agent.ts +++ b/packages/core/src/event-bus/events-agent.ts @@ -94,6 +94,16 @@ export interface AgentEvents { timestamp: number; /** Correlates start↔end for a stable activityId. */ toolCallId: string; + /** + * TRUE when the "result" is the auto-background hand-off placeholder, not a + * real outcome — the tool is still running in the background. `success` is + * true on this shape (the placeholder is a non-error result), which made the + * activity card close a still-running tool as "completed"; four reports that + * later timed out were each shown to the user as having succeeded. Consumers + * that render or count OUTCOMES must skip this emission and close on the + * `background_task:{completed,failed}` terminal event instead. + */ + backgrounded?: boolean; userId?: string; traceId?: string; agentId?: string; diff --git a/packages/core/src/event-bus/events-infra.ts b/packages/core/src/event-bus/events-infra.ts index f125ee6bb5..56d0a96809 100644 --- a/packages/core/src/event-bus/events-infra.ts +++ b/packages/core/src/event-bus/events-infra.ts @@ -657,6 +657,12 @@ export interface InfraEvents { timestamp: number; /** {@link BACKGROUND_TASK_DISPATCH_REDELIVERY} — see the failed variant. */ dispatchRedelivery?: boolean; + /** Correlation to the ORIGINATING tool call — see the failed variant. */ + toolCallId?: string; + /** Formatted session key captured at promote time — see the failed variant. */ + sessionKey?: string; + /** Trace id captured at promote time — see the failed variant. */ + traceId?: string; }; /** Background task failed (timeout, error, or daemon restart). @@ -682,6 +688,20 @@ export interface InfraEvents { * `background_task:notified.trajectoryRecorded`. */ dispatchRedelivery?: boolean; + /** + * The ORIGINATING tool call's id, captured at promote time. + * + * The activity card keys a tool's lifecycle on `tool:`; without + * this field the terminal event could not close the activity it belongs to, + * so a backgrounded tool's card either froze on "running" or (worse) had + * already been closed "completed" at hand-off. Optional: tasks recovered + * from a pre-upgrade on-disk record have none. + */ + toolCallId?: string; + /** Formatted session key captured at promote time (activity dispatch requires it). */ + sessionKey?: string; + /** Trace id captured at promote time (activity dispatch requires it). */ + traceId?: string; }; /** @@ -772,6 +792,15 @@ export interface InfraEvents { agentId: string; outcome: "success" | "denied" | "not_found"; timestamp: number; + /** + * Correlation to the turn that caused the access, when one exists. The + * audit sink prefers these over AsyncLocalStorage (which is empty for + * boot-time reads — the reason every live `secret_access` row carried + * `traceId: null` and could not be joined to a session). Emitters running + * inside a request context should pass them; boot-time emitters omit both. + */ + sessionKey?: string; + traceId?: string; }; /** diff --git a/packages/core/src/exports/activity.ts b/packages/core/src/exports/activity.ts index 79c80ccc18..065f1450e9 100644 --- a/packages/core/src/exports/activity.ts +++ b/packages/core/src/exports/activity.ts @@ -78,6 +78,7 @@ export type { ProjectionConfig, CoalesceResult, ActivityVerbosity, + FinalDeliveryReceipt, } from "../activity/index.js"; // ExecutionPlanPort — read-only SEP accessor (rides on the activity surface). diff --git a/packages/core/src/security/secret-detection.test.ts b/packages/core/src/security/secret-detection.test.ts index e2548c2081..22b44a8c99 100644 --- a/packages/core/src/security/secret-detection.test.ts +++ b/packages/core/src/security/secret-detection.test.ts @@ -163,43 +163,60 @@ describe("isSecretFieldName — superset", () => { }); // --------------------------------------------------------------------------- - // PINNED KNOWN GAP — the end-anchor hole. - // - // SECRET_FIELD_PATTERN is END-ANCHORED (`/^(.*token|.*secret|…)$/i`), so a - // credential name with ANY suffix after the keyword is not flagged BY NAME: - // - // AWS_BEARER_TOKEN_BEDROCK → false (a live provider credential) - // SECRETS_MASTER_KEY → false (the master encryption key) - // SECRET_KEY_OLD → false - // MY_TOKEN_V2 → false - // - // These are NOT necessarily leaks: `looksLikeSecretValue` is the other half of - // the defense and catches a real high-entropy credential by VALUE. But the - // field-name half has a real hole, and widening the pattern to `.*token.*` is - // exactly the "obvious security fix is often wrong" trap — it would sweep in - // `max_tokens_to_sample`, `token_count_by_model`, `tokenizer_config`, … so the - // over-match exception set would have to grow unboundedly. + // --------------------------------------------------------------------------- + // The end-anchor hole — CLOSED by the keyword-boundary matcher. // - // This test pins the CURRENT behaviour deliberately so the gap is visible and a - // half-fix fails loudly. Closing it properly needs a keyword-BOUNDARY matcher - // (split on separators/camel-case, then test each segment) rather than a suffix - // matcher — widening the pattern to `.*token.*` would sweep in - // `max_tokens_to_sample`, `token_count_by_model`, `tokenizer_config`, …. + // The suffix-anchored pattern missed any credential keyword with a suffix + // after it. The boundary matcher splits the name into word segments + // (separators + camelCase) and matches on credential segments, with the two + // AMBIGUOUS words (`token`, `key`) counting only when adjacent to a + // qualifying word — which is what closes the hole WITHOUT the `.*token.*` + // widening that would false-redact counting vocabulary (the `range_token` + // incident class). // --------------------------------------------------------------------------- - it("PINS the end-anchor gap: a credential name with a suffix is not flagged BY NAME", () => { + it("flags credential keywords MID-NAME (the closed end-anchor hole)", () => { for (const name of [ - "AWS_BEARER_TOKEN_BEDROCK", - "SECRETS_MASTER_KEY", - "SECRET_KEY_OLD", - "MY_TOKEN_V2", + "AWS_BEARER_TOKEN_BEDROCK", // bearer+token, mid-name + "SECRETS_MASTER_KEY", // "secrets" + master+key + "SECRET_KEY_OLD", // "secret", suffix after it + "OAUTH_TOKEN_V2", // oauth+token with a version suffix + "apiKeyRotationSchedule", // camelCase api+key mid-name + "DB_PASSWORD_OLD", // db+password with a suffix + "clientSecretBackup", // client+secret mid-name ]) { - expect( - isSecretFieldName(name), - `${name}: if this now returns true, the end-anchor gap was addressed — move it into the positive matrix above`, - ).toBe(false); + expect(isSecretFieldName(name), `${name} MUST be flagged`).toBe(true); } }); + it("still does NOT flag counting/selector vocabulary (the false-redaction fence)", () => { + for (const name of [ + "max_tokens_to_sample", + "token_count_by_model", + "tokenizer_config", + "range_token", + "primary_key", // database vocabulary — `key` needs a credential qualifier + "foreign_key", + "keyboard_layout", + "monkey_patch", + // META names — flags/components ABOUT credentials, never credentials. + // Redacting these corrupts config booleans and audit fields. + "writeSecretGuard", // the stock security.writeSecretGuard flag (bit live) + "secretName", + "passwordResetRequired", + "secretsStoreAvailable", + ]) { + expect(isSecretFieldName(name), `${name} must NOT be flagged`).toBe(false); + } + }); + + it("residual known miss, pinned: a bare unqualified `token` segment is not flagged", () => { + // MY_TOKEN_V2 → [my, token, v2]: "my" is not a credential qualifier. Flagging + // every bare `token` segment is exactly the over-match that false-redacted + // `range_token`. Accepted residual — the value-entropy half still covers a + // real credential stored under such a name. + expect(isSecretFieldName("MY_TOKEN_V2")).toBe(false); + }); + it("flags pattern-matched secret field names (botToken, appSecret, …)", () => { for (const name of [ "botToken", diff --git a/packages/core/src/security/secret-detection.ts b/packages/core/src/security/secret-detection.ts index 3e9e779311..92bf8a3e73 100644 --- a/packages/core/src/security/secret-detection.ts +++ b/packages/core/src/security/secret-detection.ts @@ -285,19 +285,98 @@ const NON_SECRET_TOKEN_FIELD_NAMES: ReadonlySet = new Set([ "tokenizer", ]); +// ── Keyword-boundary matcher (closes the end-anchor hole) ── + +/** + * Segments whose bare presence marks a credential name. Plurals of the + * unambiguous keywords are included (`secrets`, `passwords`, `credentials`); + * `tokens`/`keys` are NOT — the plural token family is counting vocabulary + * (`max_tokens`, `total_tokens`) and `keys` is a collection name. + */ +/** + * Single segments whose bare presence marks a credential name. Deliberately + * TINY: bare `secret`/`password`/`token`/`key` segments appear constantly in + * META names — flags and components ABOUT credentials, not credentials + * (`writeSecretGuard`, `secretName`, `passwordResetRequired`, `tokenizer`) — + * and a name-half false positive is not harmless: it REDACTS the value, which + * corrupts tool contracts and config (the `range_token` incident class; adding + * bare `secret` here false-flagged the stock `security.writeSecretGuard` + * boolean within one test cycle). `apikey` is the one collapsed word specific + * enough to stand alone. + */ +const CREDENTIAL_SEGMENTS: ReadonlySet = new Set(["apikey"]); + +/** + * Adjacent-pair rules: an AMBIGUOUS keyword counts only when the segment + * BEFORE it gives it credential meaning. This closes the END-ANCHOR hole + * (`AWS_BEARER_TOKEN_BEDROCK` → `bearer`+`token` mid-name; + * `SECRETS_MASTER_KEY` → `master`+`key`) without the `.*token.*` widening + * that would false-redact counting vocabulary, and without bare-keyword + * matches that would false-redact meta names. + */ +const TOKEN_QUALIFIERS: ReadonlySet = new Set([ + "bearer", "auth", "access", "refresh", "session", "gateway", "bot", "id", + "api", "oauth", "csrf", "jwt", +]); +const KEY_QUALIFIERS: ReadonlySet = new Set([ + "api", "private", "master", "signing", "encryption", "secret", +]); +const SECRET_QUALIFIERS: ReadonlySet = new Set([ + "client", "app", "webhook", "hmac", "signing", +]); +const PASSWORD_QUALIFIERS: ReadonlySet = new Set([ + "user", "db", "database", "admin", "root", "smtp", "proxy", "vendor", +]); + +function fieldNameSegments(name: string): string[] { + return name + .replace(/([a-z0-9])([A-Z])/g, "$1 $2") + .split(/[_\-.\s]+/) + .map((seg) => seg.toLowerCase()) + .filter((seg) => seg.length > 0); +} + +/** + * True when the segmented field name carries a credential keyword — including + * MID-NAME, which the end-anchored pattern structurally missed + * (`AWS_BEARER_TOKEN_BEDROCK`, `SECRETS_MASTER_KEY`, `SECRET_KEY_OLD`). + */ +function hasCredentialSegments(name: string): boolean { + const segs = fieldNameSegments(name); + for (let i = 0; i < segs.length; i += 1) { + const seg = segs[i]!; + if (CREDENTIAL_SEGMENTS.has(seg)) return true; + if (i === 0) continue; + const prev = segs[i - 1]!; + if (seg === "token" && TOKEN_QUALIFIERS.has(prev)) return true; + if (seg === "key" && KEY_QUALIFIERS.has(prev)) return true; + if (seg === "secret" && SECRET_QUALIFIERS.has(prev)) return true; + if (seg === "password" && PASSWORD_QUALIFIERS.has(prev)) return true; + } + return false; +} + /** * True if a FIELD NAME implies a secret: - * `SECRET_FIELD_PATTERN` ∪ the header set above, case-insensitive, MINUS the - * audited {@link NON_SECRET_TOKEN_FIELD_NAMES} over-match exceptions. + * `SECRET_FIELD_PATTERN` ∪ the header set ∪ the keyword-BOUNDARY matcher, + * case-insensitive, MINUS the audited {@link NON_SECRET_TOKEN_FIELD_NAMES} + * over-match exceptions. * * Order matters: the exception set is consulted FIRST and only ever REMOVES a - * `.*token` wildcard match — an exact header name (`x-auth-token`) is not in the - * exception set, so the header superset is unaffected. + * match — an exact header name (`x-auth-token`) is not in the exception set, + * so the header superset is unaffected. The boundary matcher closes the + * end-anchor hole (a credential keyword with a SUFFIX after it was invisible + * to the `$`-anchored pattern) without the wildcard widening that would + * false-redact counting vocabulary. */ export function isSecretFieldName(name: string): boolean { const lower = name.toLowerCase(); if (NON_SECRET_TOKEN_FIELD_NAMES.has(lower)) return false; - return SECRET_FIELD_PATTERN.test(name) || SECRET_FIELD_NAMES.has(lower); + return ( + SECRET_FIELD_PATTERN.test(name) + || SECRET_FIELD_NAMES.has(lower) + || hasCredentialSegments(name) + ); } // ── Env-ref exemption (reuses the EXPORTED env-substitution patterns) ── diff --git a/packages/daemon/src/api/obs-handlers/system-findings.ts b/packages/daemon/src/api/obs-handlers/system-findings.ts index d4a019afc4..1d47ab8d2c 100644 --- a/packages/daemon/src/api/obs-handlers/system-findings.ts +++ b/packages/daemon/src/api/obs-handlers/system-findings.ts @@ -650,7 +650,7 @@ export function buildFindings( code: "learning_health", detail: `${learningHealth.length} reflection run(s) in the window; latest outcome=${latestOutcome}, admitted=${admittedSum}, untrustedDrops=${untrustedSum}`, count: learningHealth.length, - hint: 'admitted=0 with untrustedDrops/uncorroborated is the anti-poison gates WORKING (not a fault); admitted=0 DESPITE genuine corroboration ⇒ a topicKey under-merge. Use `comis cron runs "Memory review" --agent ` for the per-run funnel; these summed counts are the deployment-wide surface.', + hint: 'admitted=0 with untrustedDrops/uncorroborated is the anti-poison gates WORKING (not a fault); admitted=0 DESPITE genuine corroboration ⇒ a topicKey under-merge. Use `comis cron runs "Memory review" --agent ` for the per-run funnel; these summed counts are the deployment-wide surface. If outcome COVERAGE is low, also check the boot log for `outcome judge unavailable` / `correction detector unavailable` — without a cheap-model key both silently no-op while learning events keep firing, so the funnel starves upstream of admission.', }); } diff --git a/packages/daemon/src/observability/obs-audit-sink.test.ts b/packages/daemon/src/observability/obs-audit-sink.test.ts new file mode 100644 index 0000000000..e0b9bfda4f --- /dev/null +++ b/packages/daemon/src/observability/obs-audit-sink.test.ts @@ -0,0 +1,67 @@ +// SPDX-License-Identifier: Apache-2.0 +/** + * The durable security-audit sink — `secret_access` correlation + trail purity. + */ +import { describe, it, expect } from "vitest"; +import { TypedEventBus } from "@comis/core"; +import { wireAuditSink } from "./obs-audit-sink.js"; +import type { AuditEventRow } from "./obs-audit-sink.js"; + +/** Wire the sink over a real bus with a capturing buffer (sqlite-only, no fs). */ +function makeSinkHarness() { + const bus = new TypedEventBus(); + const rows: AuditEventRow[] = []; + wireAuditSink({ + eventBus: bus, + auditBuffer: { push: (row: AuditEventRow) => rows.push(row) } as never, + auditConfig: { persist: true, sink: "sqlite" }, + }); + return { bus, rows }; +} + +describe("secret_access correlation + trail purity", () => { + // Live: every row carried tenantId:"" and traceId:null (boot reads have no + // ALS context), so an access could never be joined to a session — and PATH + // was audited as a secret access, diluting the trail. + function payload(over: Record = {}) { + return { + secretName: "PROVIDER_API_KEY", + agentId: "default", + outcome: "success" as const, + timestamp: 1, + ...over, + }; + } + + it("prefers the payload's correlation over AsyncLocalStorage", () => { + const { bus, rows } = makeSinkHarness(); + bus.emit("secret:accessed", payload({ traceId: "trace-7", sessionKey: "sk-7" }) as never); + expect(rows).toHaveLength(1); + expect(rows[0]!.traceId).toBe("trace-7"); + const refs = JSON.parse(String(rows[0]!.refs)); + expect(refs.origin).toBe("request"); + expect(refs.sessionKey).toBe("sk-7"); + }); + + it("tags a context-less read as origin:boot — no-context-by-design, not context-lost", () => { + const { bus, rows } = makeSinkHarness(); + bus.emit("secret:accessed", payload() as never); + expect(rows).toHaveLength(1); + const refs = JSON.parse(String(rows[0]!.refs)); + expect(refs.origin).toBe("boot"); + }); + + it("does NOT record routine environment names as secret accesses", () => { + const { bus, rows } = makeSinkHarness(); + for (const name of ["PATH", "HOME", "LANG", "TERM"]) { + bus.emit("secret:accessed", payload({ secretName: name }) as never); + } + expect(rows).toHaveLength(0); + }); + + it("STILL records a real credential read (the filter must not over-match)", () => { + const { bus, rows } = makeSinkHarness(); + bus.emit("secret:accessed", payload({ secretName: "AWS_BEARER_TOKEN_BEDROCK" }) as never); + expect(rows).toHaveLength(1); + }); +}); diff --git a/packages/daemon/src/observability/obs-audit-sink.ts b/packages/daemon/src/observability/obs-audit-sink.ts index 11d09f359f..9883436a16 100644 --- a/packages/daemon/src/observability/obs-audit-sink.ts +++ b/packages/daemon/src/observability/obs-audit-sink.ts @@ -361,6 +361,18 @@ export function wireAuditSink(deps: WireAuditSinkDeps): void { persistAuditRow(auditEventToRow(payload, tenant, agent, ctx?.traceId)); }); +/** + * Process-environment names that are configuration, not credentials. A + * `secret:accessed` for one of these is a routine env read and never belongs + * in the security access trail. Exact names only — a miss here merely leaves a + * noisy row, while an over-match would HIDE a real access. + */ +const NON_CREDENTIAL_ENV_NAMES: ReadonlySet = new Set([ + "PATH", "HOME", "LANG", "LC_ALL", "PWD", "SHELL", "TERM", "USER", + "HOSTNAME", "TZ", "TMPDIR", "NODE_ENV", +]); + + // secret:accessed — already content-free (NAME + outcome, no value). eventBus.on("secret:accessed", (payload) => { // A `not_found` READ accessed NOTHING — a routine per-turn probe for an @@ -371,17 +383,32 @@ export function wireAuditSink(deps: WireAuditSinkDeps): void { // (a secret WAS read) and the security signal keeps `denied` (unauthorized // probing is a denial, not a not_found). Mutations audit via `audit:event`. if (payload.outcome === "not_found") return; + // Routine process-environment names are not credentials; recording them as + // secret_access diluted the trail (a live box audited `PATH` as a secret + // access) and buried the env.set/mcp.connect signal the trail exists for. + if (NON_CREDENTIAL_ENV_NAMES.has(payload.secretName)) return; const ctx = tryGetContext(); + // Prefer the payload's promote-time correlation over AsyncLocalStorage — + // boot-time reads have no ALS context, which is why every live row carried + // traceId:null. `origin` makes "no context BY DESIGN" (a boot read) + // distinguishable from "context lost" (a request read that failed to + // correlate) — previously both looked identically blank. + const traceId = payload.traceId ?? ctx?.traceId; persistAuditRow(buildAuditRow({ kind: "secret_access", ts: payload.timestamp, tenant: ctx?.tenantId ?? "", agent: payload.agentId, - traceId: ctx?.traceId, + traceId, action: payload.secretName, actor: payload.agentId, outcome: payload.outcome, - refs: { secretName: payload.secretName, outcome: payload.outcome }, + refs: { + secretName: payload.secretName, + outcome: payload.outcome, + origin: traceId !== undefined || payload.sessionKey !== undefined ? "request" : "boot", + ...(payload.sessionKey !== undefined ? { sessionKey: payload.sessionKey } : {}), + }, })); }); diff --git a/packages/observability/src/activity/activity-stream.test.ts b/packages/observability/src/activity/activity-stream.test.ts index 674484a319..d951d880d4 100644 --- a/packages/observability/src/activity/activity-stream.test.ts +++ b/packages/observability/src/activity/activity-stream.test.ts @@ -848,3 +848,128 @@ describe("activity-stream — markers.running prefix on phase:start", () => { sub.unsubscribe(); }); }); + +describe("backgrounded tools close on their REAL terminal, not the hand-off (F-ACT-1)", () => { + // Proven live: four backgrounded reports each closed "completed" at hand-off; + // every one subsequently failed with a 120s MCP timeout. The card told the + // user four reports succeeded, and the turn then finalized failure with + // failedEventCount:0 — nothing to name — rendering a failure pill above the + // eventually-delivered answer. + const CTX = { + sessionKey: "default:agent:default:u1:telegram:peer:u1", + agentId: "default", + traceId: "trace-1", + }; + + function harness() { + const bus = new TypedEventBus(); + const stream = createActivityStream({ eventBus: bus }); + const events: ActivityEvent[] = []; + const sub = stream.subscribeForTurn( + { sessionKey: CTX.sessionKey, agentId: CTX.agentId, traceId: CTX.traceId }, + (e) => events.push(e), + ); + return { bus, stream, events, sub }; + } + + function toolExecuted(over: Record = {}) { + return { + toolName: "mcp__vendor-mcp--vendor_activity_report", + toolCallId: "call-1", + durationMs: 10_000, + success: true, + timestamp: 1, + ...CTX, + ...over, + }; + } + + it("a hand-off (backgrounded:true) does NOT close the activity", () => { + const { bus, events } = harness(); + bus.emit("tool:executed", toolExecuted({ backgrounded: true }) as never); + const ends = events.filter((e) => e.phase === "end"); + expect(ends).toHaveLength(0); + }); + + it("the background_task:failed terminal closes it FAILED with the correlated id", () => { + const { bus, events } = harness(); + bus.emit("tool:executed", toolExecuted({ backgrounded: true }) as never); + bus.emit("background_task:failed", { + agentId: CTX.agentId, + taskId: "task-1", + toolName: "mcp__vendor-mcp--vendor_activity_report", + error: "MCP error -32001: Request timed out", + durationMs: 120_000, + origin: {} as never, + timestamp: 2, + toolCallId: "call-1", + sessionKey: CTX.sessionKey, + traceId: CTX.traceId, + } as never); + const ends = events.filter((e) => e.phase === "end"); + expect(ends).toHaveLength(1); + expect(ends[0]!.status).toBe("failed"); + expect(ends[0]!.toolCallId).toBe("call-1"); + expect(ends[0]!.errorKind).toBe("dependency"); + }); + + it("the background_task:completed terminal closes it COMPLETED", () => { + const { bus, events } = harness(); + bus.emit("tool:executed", toolExecuted({ backgrounded: true }) as never); + bus.emit("background_task:completed", { + agentId: CTX.agentId, + taskId: "task-1", + toolName: "mcp__vendor-mcp--vendor_activity_report", + durationMs: 60_000, + origin: {} as never, + timestamp: 2, + toolCallId: "call-1", + sessionKey: CTX.sessionKey, + traceId: CTX.traceId, + } as never); + const ends = events.filter((e) => e.phase === "end"); + expect(ends).toHaveLength(1); + expect(ends[0]!.status).toBe("completed"); + }); + + it("a dispatchRedelivery re-emit does NOT re-close (one outcome per task)", () => { + const { bus, events } = harness(); + const failed = { + agentId: CTX.agentId, + taskId: "task-1", + toolName: "report", + error: "boom", + durationMs: 5, + origin: {} as never, + timestamp: 2, + toolCallId: "call-1", + sessionKey: CTX.sessionKey, + traceId: CTX.traceId, + }; + bus.emit("background_task:failed", failed as never); + bus.emit("background_task:failed", { ...failed, dispatchRedelivery: true } as never); + expect(events.filter((e) => e.phase === "end")).toHaveLength(1); + }); + + it("a pre-upgrade terminal with NO toolCallId is a no-op (the old close already happened)", () => { + const { bus, events } = harness(); + bus.emit("background_task:failed", { + agentId: CTX.agentId, + taskId: "task-1", + toolName: "report", + error: "boom", + durationMs: 5, + origin: {} as never, + timestamp: 2, + } as never); + expect(events.filter((e) => e.phase === "end")).toHaveLength(0); + }); + + it("a NON-backgrounded tool:executed still closes exactly as before (regression fence)", () => { + const { bus, events } = harness(); + bus.emit("tool:executed", toolExecuted() as never); + const ends = events.filter((e) => e.phase === "end"); + expect(ends).toHaveLength(1); + expect(ends[0]!.status).toBe("completed"); + }); +}); diff --git a/packages/observability/src/activity/activity-stream.ts b/packages/observability/src/activity/activity-stream.ts index 35a446823d..249535bc8c 100644 --- a/packages/observability/src/activity/activity-stream.ts +++ b/packages/observability/src/activity/activity-stream.ts @@ -155,6 +155,8 @@ const SUBSCRIBED_EVENTS = [ "tool:started", "tool:executed", "tool:timeout", + "background_task:completed", + "background_task:failed", "model:fallback_attempt", "model:fallback_exhausted", "model:lkw_fallback_attempt", @@ -462,6 +464,13 @@ export function createActivityStream(deps: CreateActivityStreamDeps): ActivitySt function onToolExecuted(p: EventMap["tool:executed"]): void { if (isSuppressed(p.toolName)) return; + // A background hand-off is NOT an outcome — the tool is still running and + // p.success is true only because the placeholder is a non-error result. + // Leave the activity running; the background_task:{completed,failed} + // terminal event closes it (onBackgroundTaskTerminal below). Closing here + // told the user four still-running reports had "completed" — every one of + // which subsequently failed. + if (p.backgrounded === true) return; if (p.agentId === undefined || p.sessionKey === undefined || p.traceId === undefined) return; const { defaultLabel, semanticPhase } = buildLabel(p.toolName, undefined, p.params ?? {}); dispatch({ @@ -483,6 +492,43 @@ export function createActivityStream(deps: CreateActivityStreamDeps): ActivitySt }); } + /** + * Close a BACKGROUNDED tool's activity on its real terminal event. + * + * The hand-off left the card `running` (see onToolExecuted); this is the + * matching close. Correlation is the promote-time capture on the event — + * a pre-upgrade task record has none, in which case there is no card to + * close (the old hand-off path already closed it) and we do nothing. + * A `dispatchRedelivery` re-emit is a dispatch mechanism, not a second + * outcome — never re-close on it. + */ + function onBackgroundTaskTerminal( + p: EventMap["background_task:completed"] | EventMap["background_task:failed"], + failed: boolean, + ): void { + if (p.dispatchRedelivery === true) return; + if (p.toolCallId === undefined || p.sessionKey === undefined) return; + if (isSuppressed(p.toolName)) return; + const { defaultLabel, semanticPhase } = buildLabel(p.toolName, undefined, {}); + dispatch({ + schemaVersion: 1, + activityId: activityIdFor(`tool:${p.toolCallId}`), + sessionKey: p.sessionKey, + agentId: p.agentId, + traceId: p.traceId ?? p.taskId, + toolCallId: p.toolCallId, + ts: ts(), + phase: "end", + status: failed ? "failed" : "completed", + kind: "tool", + semanticPhase: failed ? "error" : semanticPhase, + toolName: p.toolName, + durationMs: p.durationMs, + ...(failed ? { errorKind: "dependency" as const } : {}), + defaultLabel, + }); + } + function onToolTimeout(p: EventMap["tool:timeout"]): void { if (isSuppressed(p.toolName)) return; if (p.agentId === undefined || p.sessionKey === undefined) return; @@ -656,6 +702,8 @@ export function createActivityStream(deps: CreateActivityStreamDeps): ActivitySt bind("tool:started", onToolStarted); bind("tool:executed", onToolExecuted); bind("tool:timeout", onToolTimeout); + bind("background_task:completed", (p) => onBackgroundTaskTerminal(p, false)); + bind("background_task:failed", (p) => onBackgroundTaskTerminal(p, true)); bind("model:fallback_attempt", onModelEvent); bind("model:fallback_exhausted", onModelEvent); bind("model:lkw_fallback_attempt", onModelEvent); diff --git a/packages/orchestrator/src/execution/activity-turn-coordinator.test.ts b/packages/orchestrator/src/execution/activity-turn-coordinator.test.ts index 2a99803391..390218dbca 100644 --- a/packages/orchestrator/src/execution/activity-turn-coordinator.test.ts +++ b/packages/orchestrator/src/execution/activity-turn-coordinator.test.ts @@ -1266,3 +1266,76 @@ describe("a failure with no failed activity events never renders a naked errorKi coord.dispose(); }); }); + +describe("a delivered answer never renders a failure pill (F-ACT-1 layer 4)", () => { + // Proven live on the fixed-hint build: the Excel turn DELIVERED its artifact + // and the card still showed "❌ dependency — a step failed outside the tool + // timeline" above it; the agent then explained the pill away unprompted. + it("failure + delivered evidence + observed failed events → success_with_recovered_failures", async () => { + const clock = createFakeClock(5_000); + const { deps, timer, stream, renderer } = makeCoordinatorDeps({ clock }); + const coord = createActivityTurnCoordinator(deps); + coord.start(makeCtx()); + + // The backgrounded tool's REAL terminal, observed during the turn. + stream.emit(makeEvent({ status: "failed", errorKind: "dependency", phase: "end" })); + timer.advance(800); + + await coord.finalize({ + kind: "failure", + errorKind: "dependency", + failedEvents: [], + delivery: { deliveredAtMs: clock.now() } as never, + }); + + const outcome = renderer.finalizeCalls[0]!.outcome; + expect(outcome.kind).toBe("success_with_recovered_failures"); + // The failure evidence is PRESERVED, not erased. + const recovered = (outcome as { recoveredFailures: readonly unknown[] }).recoveredFailures; + expect(recovered.length).toBeGreaterThan(0); + coord.dispose(); + }); + + it("failure + delivered evidence but NO observed failed events keeps the truthful failure", async () => { + const clock = createFakeClock(5_000); + const { deps, renderer } = makeCoordinatorDeps({ clock }); + const coord = createActivityTurnCoordinator(deps); + coord.start(makeCtx()); + + await coord.finalize({ + kind: "failure", + errorKind: "dependency", + failedEvents: [], + delivery: { deliveredAtMs: clock.now() } as never, + }); + + // No events to attribute — reclassifying would ERASE the failure (the + // false-success trap). It stays a failure, with the named reason. + const outcome = renderer.finalizeCalls[0]!.outcome; + expect(outcome.kind).toBe("failure"); + expect((outcome as { reason?: string }).reason).toBeDefined(); + coord.dispose(); + }); + + it("a resource abort with delivery evidence NEVER reclassifies (stopped renders as stopped)", async () => { + const clock = createFakeClock(5_000); + const { deps, timer, stream, renderer } = makeCoordinatorDeps({ clock }); + const coord = createActivityTurnCoordinator(deps); + coord.start(makeCtx()); + stream.emit(makeEvent({ status: "failed", errorKind: "resource", phase: "end" })); + timer.advance(800); + + // withDeliveredEvidence never attaches delivery to a reasoned abort, so the + // coordinator sees a plain failure — but pin the coordinator side too: + await coord.finalize({ + kind: "failure", + errorKind: "resource", + failedEvents: [], + reason: "stopped — spend limit reached", + }); + const outcome = renderer.finalizeCalls[0]!.outcome; + expect(outcome.kind).toBe("failure"); + expect((outcome as { reason?: string }).reason).toBe("stopped — spend limit reached"); + coord.dispose(); + }); +}); diff --git a/packages/orchestrator/src/execution/activity-turn-coordinator.ts b/packages/orchestrator/src/execution/activity-turn-coordinator.ts index d5cb77bcb6..4c1af51bed 100644 --- a/packages/orchestrator/src/execution/activity-turn-coordinator.ts +++ b/packages/orchestrator/src/execution/activity-turn-coordinator.ts @@ -575,6 +575,25 @@ export function createActivityTurnCoordinator(deps: ActivityTurnCoordinatorDeps) } } + // (1a2) A failure whose answer WAS fully delivered, with the failure + // attributable to observed events, is a SUCCESS WITH RECOVERED FAILURES: + // the user's chat shows the answer (no failure pill), while the failed + // events ride the outcome + `activity:turn_finalized` as evidence. This is + // NOT "delivery succeeded ⇒ success" — with no observed failed events the + // failure evidence would be erased, so that case keeps the truthful + // failure (and gets the named reason below). + if (effective.kind === "failure" && effective.delivery !== undefined) { + const failedEvents = events.filter((e) => e.status === "failed"); + if (isNonEmptyEvents(failedEvents)) { + effective = { + kind: "success_with_recovered_failures", + trivial: false, + delivery: effective.delivery, + recoveredFailures: failedEvents, + }; + } + } + // (1b) An unattributed FAILURE must still read truthfully. A `failure` with // zero failed events cannot name a tool, so `failureLabel()` renders the bare // "❌ {errorKind}" — which reached a real user as a chat bubble whose entire diff --git a/packages/orchestrator/src/execution/execution-pipeline.ts b/packages/orchestrator/src/execution/execution-pipeline.ts index 0330554938..170937ca66 100644 --- a/packages/orchestrator/src/execution/execution-pipeline.ts +++ b/packages/orchestrator/src/execution/execution-pipeline.ts @@ -49,7 +49,7 @@ import { import { emitObservationalEvent } from "./execution-event-emitter.js"; import { createMediaDeliveryFailureReceipt } from "./execution-media-receipt.js"; import { runExecutionPolicy } from "./execution-policy.js"; -import { mapAbortToTurnOutcome } from "./turn-outcome-mapper.js"; +import { mapAbortToTurnOutcome, withDeliveredEvidence } from "./turn-outcome-mapper.js"; import { classifyExecutionAbortReason, classifyExecutionFinishReason, @@ -703,9 +703,17 @@ export async function executeAndDeliver( // renderer/coordinator failure is contained by finalizeCoordinator so it // cannot reclassify an already-delivered turn or trigger inbound fallback. if (coordinatorExecutionOutcome) { - // Partial text may have delivered, but the run was stopped — render the - // truthful failure, not a success. - await finalizeCoordinator(coordinatorExecutionOutcome); + // A resource abort (reason set) renders as the truthful stop even when + // partial text delivered. An UNATTRIBUTED lifecycle failure whose final + // answer WAS fully delivered gets the receipt attached, so the + // coordinator can reclassify to success_with_recovered_failures instead + // of pinning a failure marker above a delivered answer (proven live). + await finalizeCoordinator( + withDeliveredEvidence( + coordinatorExecutionOutcome, + deliveryReceipt.ok ? deliveryReceipt.value : undefined, + ), + ); } else if (mediaFailureReceipt !== undefined) { await finalizeCoordinator({ kind: "failure", diff --git a/packages/orchestrator/src/execution/turn-outcome-mapper.test.ts b/packages/orchestrator/src/execution/turn-outcome-mapper.test.ts index 948a4524f2..87bb34246a 100644 --- a/packages/orchestrator/src/execution/turn-outcome-mapper.test.ts +++ b/packages/orchestrator/src/execution/turn-outcome-mapper.test.ts @@ -11,7 +11,7 @@ */ import { describe, it, expect } from "vitest"; -import { mapAbortToTurnOutcome } from "./turn-outcome-mapper.js"; +import { mapAbortToTurnOutcome, withDeliveredEvidence } from "./turn-outcome-mapper.js"; describe("mapAbortToTurnOutcome", () => { it("maps a max_steps resource abort to a truthful resource failure mentioning the step limit", () => { @@ -80,3 +80,44 @@ describe("mapAbortToTurnOutcome", () => { expect(outcome.failedEvents).toEqual([]); }); }); + +describe("withDeliveredEvidence", () => { + const receipt = { deliveredAtMs: 1000 } as never; + + it("attaches the receipt to an UNATTRIBUTED failure", () => { + const out = withDeliveredEvidence( + { kind: "failure", errorKind: "dependency", failedEvents: [] }, + receipt, + ); + expect(out.kind).toBe("failure"); + expect((out as { delivery?: unknown }).delivery).toBe(receipt); + }); + + it("NEVER attaches to a resource abort — a stopped run renders as stopped", () => { + const out = withDeliveredEvidence( + { kind: "failure", errorKind: "resource", failedEvents: [], reason: "stopped — hit step limit" }, + receipt, + ); + expect((out as { delivery?: unknown }).delivery).toBeUndefined(); + }); + + it("never attaches to a DELIVERY failure (deliveryReceipt present)", () => { + const out = withDeliveredEvidence( + { kind: "failure", errorKind: "platform", failedEvents: [], deliveryReceipt: {} as never }, + receipt, + ); + expect((out as { delivery?: unknown }).delivery).toBeUndefined(); + }); + + it("passes success/aborted outcomes through untouched", () => { + const success = { kind: "success", trivial: false, delivery: receipt } as const; + expect(withDeliveredEvidence(success as never, receipt)).toBe(success); + const aborted = { kind: "aborted", reason: "timeout" } as const; + expect(withDeliveredEvidence(aborted as never, receipt)).toBe(aborted); + }); + + it("no receipt (delivery failed) → unchanged", () => { + const failure = { kind: "failure", errorKind: "dependency", failedEvents: [] } as const; + expect(withDeliveredEvidence(failure as never, undefined)).toBe(failure); + }); +}); diff --git a/packages/orchestrator/src/execution/turn-outcome-mapper.ts b/packages/orchestrator/src/execution/turn-outcome-mapper.ts index 0a2baf0060..6ddff8ecd8 100644 --- a/packages/orchestrator/src/execution/turn-outcome-mapper.ts +++ b/packages/orchestrator/src/execution/turn-outcome-mapper.ts @@ -15,7 +15,7 @@ * carry no raw provider/internal text — only the closed-union errorKind + * fixed copy reach the render path (information-disclosure guard, T-hbe-02). */ -import type { TurnOutcome } from "@comis/core"; +import type { TurnOutcome, FinalDeliveryReceipt } from "@comis/core"; /** One-line human reason for a step-limit (max_steps) abort. */ const STEP_LIMIT_REASON = "stopped — hit step limit" as const; @@ -70,3 +70,30 @@ export function mapAbortToTurnOutcome(input: AbortSignalInput): TurnOutcome | un reason, }; } + +/** + * Attach delivered-answer evidence to an UNATTRIBUTED execution failure. + * + * Pure. Returns the outcome unchanged unless ALL hold: + * - it is a `failure`, + * - with NO `reason` (a resource abort — step limit / loop / spend — must render + * as stopped even when partial text delivered; its reason is the honest label), + * - with NO deliveryReceipt (a delivery failure is a real failure), and + * - the delivery genuinely succeeded (`receipt` present). + * + * The coordinator uses the attached receipt to reclassify the turn to + * `success_with_recovered_failures` when it also observed failed activity + * events — the user sees their answer without a failure pill while the failure + * stays on the outcome and the `activity:turn_finalized` event. Without this, + * a delivered answer rendered with "❌ dependency" above it (proven live). + */ +export function withDeliveredEvidence( + outcome: TurnOutcome, + receipt: FinalDeliveryReceipt | undefined, +): TurnOutcome { + if (receipt === undefined) return outcome; + if (outcome.kind !== "failure") return outcome; + if (outcome.reason !== undefined && outcome.reason.length > 0) return outcome; + if (outcome.deliveryReceipt !== undefined) return outcome; + return { ...outcome, delivery: receipt }; +} diff --git a/packages/skills/src/__tests__/__snapshots__/tool-registry.typebox-1.3.6.snap b/packages/skills/src/__tests__/__snapshots__/tool-registry.typebox-1.3.6.snap index e5b3bd1dad..c80a7f06ed 100644 --- a/packages/skills/src/__tests__/__snapshots__/tool-registry.typebox-1.3.6.snap +++ b/packages/skills/src/__tests__/__snapshots__/tool-registry.typebox-1.3.6.snap @@ -1098,6 +1098,10 @@ "description": "Maximum results. For history: default 10. For env_list: default 100, max 500 (clamped).", "type": "number" }, + "pending_action_id": { + "description": "The pending_action_id returned by a gated env_set. Pass it back with _confirmed: true INSTEAD of re-sending env_value — the daemon replays the stashed value, so the secret never has to survive in conversation context across the approval round-trip.", + "type": "string" + }, "section": { "description": "Config section name. Required for patch/apply/schema, optional for read. Valid sections: agents, channels, memory, security, routing, daemon, scheduler, gateway, integrations, monitoring, browser, models, providers, messages, approvals. Note: MCP server settings are under 'integrations', not 'mcp'.", "type": "string" diff --git a/packages/skills/src/platform-tools/tools/gateway-tool.test.ts b/packages/skills/src/platform-tools/tools/gateway-tool.test.ts index b63d978016..c0d076bafe 100644 --- a/packages/skills/src/platform-tools/tools/gateway-tool.test.ts +++ b/packages/skills/src/platform-tools/tools/gateway-tool.test.ts @@ -637,17 +637,20 @@ describe("gateway tool", () => { ).rejects.toThrow(/Missing required parameter: env_key/); }); - it("throws when env_value parameter missing", async () => { - const rpcCall = createMockRpcCall(); + it("returns a structured error when env_value is missing (names the confirm-by-id path)", async () => { + const rpcCall = vi.fn(async (method: string) => + method === "gateway.status" + ? { pid: 1, uptime: 1, memoryUsage: 1, nodeVersion: "v22", configPaths: [], sections: [], secretsStoreAvailable: true } + : {}); const tool = createGatewayTool(rpcCall, mockLogger); - - await expect( - tool.execute("call-e5", { - action: "env_set" as "read", - env_key: "MY_KEY", - _confirmed: true, - } as any), - ).rejects.toThrow(/Missing required parameter: env_value/); + const result = await tool.execute("call-mv", { + action: "env_set" as "read", + env_key: "SOME_KEY", + } as any); + const details = result.details as Record; + expect(details.error).toBe("missing_env_value"); + // …and teaches the confirm-by-id path instead of demanding a re-send. + expect(String(details.hint)).toContain("pending_action_id"); }); it("rejects literal [REDACTED] placeholder without calling env.set", async () => { @@ -1307,3 +1310,106 @@ describe("the env_set placeholder hint must not cause credential re-transmission expect(h).toMatch(/never enters the session|encrypted store/i); }); }); + +describe("gated env_set survives its own confirmation (the approval/redaction deadlock)", () => { + // Live shape: request → gate → user approves → the redaction pass has + // rewritten the secret in context → the confirm re-sent "[REDACTED]" → dead + // end, and the user was asked to re-paste their password into the chat. + // With the pending stash, the confirm replays the ORIGINAL value by id. + function statusOk() { + return { + pid: 1, uptime: 1, memoryUsage: 1, nodeVersion: "v22", configPaths: [], sections: [], + secretsStoreAvailable: true, + }; + } + + it("the gate returns a pending_action_id and the confirm-by-id replays the ORIGINAL value", async () => { + const envSetCalls: Array> = []; + const rpcCall = vi.fn(async (method: string, params?: Record) => { + if (method === "gateway.status") return statusOk(); + if (method === "env.set") { envSetCalls.push(params!); return { set: true, key: params!.key, storage: "encrypted" }; } + return {}; + }); + const tool = createGatewayTool(rpcCall, mockLogger); + + // 1) the gated request — carries the real secret ONCE + const gated = await tool.execute("call-p1", { + action: "env_set" as "read", + env_key: "VENDOR_PASSWORD", + env_value: "the-real-secret-value", + } as any); + const gatedDetails = gated.details as Record; + expect(gatedDetails.requiresConfirmation).toBe(true); + const pendingId = gatedDetails.pending_action_id as string; + expect(typeof pendingId).toBe("string"); + expect(String(gatedDetails.hint)).toContain("OMIT env_value"); + + // 2) the confirm — NO env_value at all (context may have been redacted) + const confirmed = await tool.execute("call-p2", { + action: "env_set" as "read", + env_key: "VENDOR_PASSWORD", + _confirmed: true, + pending_action_id: pendingId, + } as any); + const confirmedDetails = confirmed.details as Record; + expect(confirmedDetails.set).toBe(true); + // the daemon received the ORIGINAL value, not a placeholder + expect(envSetCalls).toHaveLength(1); + expect(envSetCalls[0]!.value).toBe("the-real-secret-value"); + }); + + it("a confirm whose context re-sent the PLACEHOLDER still succeeds via the stash", async () => { + const envSetCalls: Array> = []; + const rpcCall = vi.fn(async (method: string, params?: Record) => { + if (method === "gateway.status") return statusOk(); + if (method === "env.set") { envSetCalls.push(params!); return { set: true }; } + return {}; + }); + const tool = createGatewayTool(rpcCall, mockLogger); + const gated = await tool.execute("call-p3", { + action: "env_set" as "read", env_key: "K", env_value: "real-value", + } as any); + const pendingId = (gated.details as Record).pending_action_id as string; + + const confirmed = await tool.execute("call-p4", { + action: "env_set" as "read", + env_key: "K", + env_value: "[REDACTED]", // what a redacted replay context would supply + _confirmed: true, + pending_action_id: pendingId, + } as any); + expect((confirmed.details as Record).set).toBe(true); + expect(envSetCalls[0]!.value).toBe("real-value"); + }); + + it("the pending id is ONE-SHOT — a second confirm replays nothing", async () => { + const rpcCall = vi.fn(async (method: string) => + method === "gateway.status" ? statusOk() : { set: true }); + const tool = createGatewayTool(rpcCall, mockLogger); + const gated = await tool.execute("call-p5", { + action: "env_set" as "read", env_key: "K2", env_value: "v", + } as any); + const pendingId = (gated.details as Record).pending_action_id as string; + await tool.execute("call-p6", { + action: "env_set" as "read", env_key: "K2", _confirmed: true, pending_action_id: pendingId, + } as any); + const second = await tool.execute("call-p7", { + action: "env_set" as "read", env_key: "K2", _confirmed: true, pending_action_id: pendingId, + } as any); + expect((second.details as Record).error).toBe("pending_action_expired"); + }); + + it("a key mismatch between stash and confirm is rejected", async () => { + const rpcCall = vi.fn(async (method: string) => + method === "gateway.status" ? statusOk() : { set: true }); + const tool = createGatewayTool(rpcCall, mockLogger); + const gated = await tool.execute("call-p8", { + action: "env_set" as "read", env_key: "RIGHT_KEY", env_value: "v", + } as any); + const pendingId = (gated.details as Record).pending_action_id as string; + const wrong = await tool.execute("call-p9", { + action: "env_set" as "read", env_key: "WRONG_KEY", _confirmed: true, pending_action_id: pendingId, + } as any); + expect((wrong.details as Record).error).toBe("pending_action_key_mismatch"); + }); +}); diff --git a/packages/skills/src/platform-tools/tools/gateway-tool.ts b/packages/skills/src/platform-tools/tools/gateway-tool.ts index f7f829dab8..d0ca2e818e 100644 --- a/packages/skills/src/platform-tools/tools/gateway-tool.ts +++ b/packages/skills/src/platform-tools/tools/gateway-tool.ts @@ -11,6 +11,7 @@ */ import type { AgentTool } from "@earendil-works/pi-agent-core"; +import { randomUUID } from "node:crypto"; import { Type } from "typebox"; import { tryGetContext, @@ -28,7 +29,7 @@ import { } from "../tool-helpers.js"; import { createMultiActionDispatchTool } from "../messaging-factory.js"; import type { RpcCall } from "./cron-tool.js"; -import { isDocker } from "@comis/core"; +import { isDocker, systemNowMs } from "@comis/core"; import type { ComisLogger } from "@comis/core"; // --------------------------------------------------------------------------- @@ -126,6 +127,15 @@ const GatewayToolParams = Type.Object({ "and after they approve, call the same action again with _confirmed: true.", }), ), + pending_action_id: Type.Optional( + Type.String({ + description: + "The pending_action_id returned by a gated env_set. Pass it back with " + + "_confirmed: true INSTEAD of re-sending env_value — the daemon replays the " + + "stashed value, so the secret never has to survive in conversation context " + + "across the approval round-trip.", + }), + ), }); /** @@ -136,6 +146,62 @@ const GatewayToolParams = Type.Object({ * is not itself the confirmation. Fixes the foot-gun where an agent, seeing an inline * pre-confirmation, replied "done" without re-calling (nothing was actually written). */ +// ── Pending gated env_set stash (the approval/redaction deadlock fix) ── +// +// An approval-gated env_set needs a SECOND call with `_confirmed: true`, but the +// session-redaction pass rewrites the secret in the originating message between +// the two calls — so by the time consent exists, the value is gone from every +// surface the agent can read, and the only way to comply was to ask the user to +// re-send the credential through the chat (observed live). The fix: the FIRST +// call stashes {key, value} server-side (this module runs in the daemon — the +// same trust domain that will store the secret anyway) and returns an opaque +// pending_action_id; the confirm passes the id back and the stashed value is +// replayed. The secret never needs to survive in conversation context, so the +// redaction pass can stay maximally aggressive. +// +// Bounded: TTL + a small cap, sweep on every touch. In-memory by design — a +// daemon restart voids pending approvals, which is the safe failure mode. + +const PENDING_ENV_SET_TTL_MS = 5 * 60 * 1000; +const PENDING_ENV_SET_MAX = 20; + +interface PendingEnvSet { + envKey: string; + envValue: string; + expiresAtMs: number; +} + +const pendingEnvSets = new Map(); + +function sweepPendingEnvSets(nowMs: number): void { + for (const [id, entry] of pendingEnvSets) { + if (entry.expiresAtMs <= nowMs) pendingEnvSets.delete(id); + } + // Oldest-first eviction beyond the cap (Map preserves insertion order). + while (pendingEnvSets.size > PENDING_ENV_SET_MAX) { + const oldest = pendingEnvSets.keys().next().value; + if (oldest === undefined) break; + pendingEnvSets.delete(oldest); + } +} + +/** Stash a gated env_set; returns the opaque id the confirm passes back. */ +export function stashPendingEnvSet(envKey: string, envValue: string, nowMs: number): string { + sweepPendingEnvSets(nowMs); + const id = randomUUID(); + pendingEnvSets.set(id, { envKey, envValue, expiresAtMs: nowMs + PENDING_ENV_SET_TTL_MS }); + return id; +} + +/** Take (and consume) a stashed env_set. One-shot: a second confirm replays nothing. */ +export function takePendingEnvSet(id: string, nowMs: number): PendingEnvSet | undefined { + sweepPendingEnvSets(nowMs); + const entry = pendingEnvSets.get(id); + if (entry === undefined) return undefined; + pendingEnvSets.delete(id); + return entry; +} + export function confirmationRequiredHint(action: string): string { return ( `NOT performed — "${action}" has NOT run and nothing has changed. Do NOT report success. ` + @@ -335,7 +401,47 @@ export function createGatewayTool( default: { // action === "env_set" const envKey = readStringParam(p, "env_key")!; - const envValue = readStringParam(p, "env_value")!; + // Optional on the confirm-by-id leg (the caller is TOLD to omit it — + // the stashed value is replayed server-side); required otherwise. + let envValue = readStringParam(p, "env_value", false); + + // CONFIRM leg: when the caller passes back the pending_action_id, + // replay the STASHED value — the one from the original request — + // regardless of what (possibly redaction-mangled) env_value the + // model still holds. One-shot + TTL'd; an expired/unknown id gets + // an honest error rather than silently writing the mangled value. + const pendingActionId = readStringParam(p, "pending_action_id", false); + if (p._confirmed === true && pendingActionId !== undefined && pendingActionId.length > 0) { + const pending = takePendingEnvSet(pendingActionId, systemNowMs()); + if (pending === undefined) { + return { + error: "pending_action_expired", + hint: + "The pending env_set was not found — it expired (5 min TTL), was already " + + "consumed, or the daemon restarted. Re-initiate the env_set from scratch; " + + "if the secret is no longer available in the conversation, have the operator " + + "run `comis secrets set ` on the host instead.", + }; + } + if (pending.envKey !== envKey) { + return { + error: "pending_action_key_mismatch", + hint: + `The pending action stashed a value for "${pending.envKey}", not "${envKey}". ` + + "Confirm the SAME action that was gated, or re-initiate.", + }; + } + envValue = pending.envValue; + } + if (envValue === undefined || envValue.length === 0) { + return { + error: "missing_env_value", + hint: + "env_set needs env_value on the first (gated) call. On the confirm leg, " + + "pass pending_action_id with _confirmed: true instead — the original value " + + "is replayed server-side.", + }; + } // Reject session-redaction placeholders. This catches the replay // poisoning loop where a prior env_set tool call's arguments were @@ -413,10 +519,21 @@ export function createGatewayTool( const gate = envSetGate(p); if (gate.requiresConfirmation) { + // Stash the request server-side so the confirm can replay it by id + // — the value must NOT need to survive in conversation context + // across the approval round-trip (the redaction pass will rewrite + // it there, and that deadlock forced a live user to re-paste a + // password into the chat). + const pendingId = stashPendingEnvSet(envKey, envValue, systemNowMs()); return { requiresConfirmation: true, actionType: gate.actionType, - hint: confirmationRequiredHint(`setting secret "${envKey}"`), + pending_action_id: pendingId, + hint: + confirmationRequiredHint(`setting secret "${envKey}"`) + + ` When re-calling, pass pending_action_id: "${pendingId}" with _confirmed: true` + + " and OMIT env_value — the original value is replayed server-side (do not" + + " re-send the secret; it may have been redacted from your context).", }; } const result = await rpcCall("env.set", { key: envKey, value: envValue, _trustLevel }); diff --git a/packages/skills/src/tools/builtin/exec-diagnostics.test.ts b/packages/skills/src/tools/builtin/exec-diagnostics.test.ts index 051f6f3626..41b083b37d 100644 --- a/packages/skills/src/tools/builtin/exec-diagnostics.test.ts +++ b/packages/skills/src/tools/builtin/exec-diagnostics.test.ts @@ -412,3 +412,27 @@ describe("matchExecRecoveryHint — workspace-rooted venv missing matcher", () = expect(result!).not.toContain("pyproject.toml"); }); }); + +describe("matchExternallyManagedEnv (PEP 668)", () => { + const PEP668 = `error: externally-managed-environment + +\u00d7 This environment is externally managed +\u2570\u2500> To install Python packages system-wide, try apt install python3-xyz + note: If you believe this is a mistake, please contact your Python installation provider. + hint: See PEP 668 for the detailed specification.`; + + it("surfaces the venv + --break-system-packages options at the head of stderr", () => { + const hint = matchExecRecoveryHint({ stderr: PEP668, exitCode: 1, cwd: "/w" }); + expect(hint).not.toBeNull(); + expect(hint).toContain("PEP 668"); + expect(hint).toContain("--break-system-packages"); + expect(hint).toContain("venv"); + // …and forbids the blind identical retry that wasted live round-trips. + expect(hint).toMatch(/Do not retry the identical command/); + }); + + it("abstains on success and on unrelated stderr", () => { + expect(matchExecRecoveryHint({ stderr: PEP668, exitCode: 0, cwd: "/w" })).toBeNull(); + expect(matchExecRecoveryHint({ stderr: "some other error", exitCode: 1, cwd: "/w" })).toBeNull(); + }); +}); diff --git a/packages/skills/src/tools/builtin/exec-diagnostics.ts b/packages/skills/src/tools/builtin/exec-diagnostics.ts index c8e8213674..28a4737b9d 100644 --- a/packages/skills/src/tools/builtin/exec-diagnostics.ts +++ b/packages/skills/src/tools/builtin/exec-diagnostics.ts @@ -193,9 +193,30 @@ const matchVenvMissing: Matcher = ({ stderr, exitCode, cwd }) => { // Registry + entry point // --------------------------------------------------------------------------- + +/** + * PEP 668: a system Python refuses `pip install` with + * `error: externally-managed-environment`. Observed live: an agent burned three + * exec round-trips (bare pip → read the wall of text → retry with + * `--break-system-packages`) before its xlsx task could start. The stderr wall + * does mention the flag, but buried after a distro lecture — surface the two + * viable next steps at the HEAD so the first retry is the right one. + */ +const matchExternallyManagedEnv: Matcher = ({ stderr, exitCode }) => { + if (exitCode === 0) return null; + if (!stderr || !stderr.includes("externally-managed-environment")) return null; + return ( + "RECOVERY HINT: This system Python is PEP 668 externally-managed — bare `pip install` " + + "will always refuse. Either install into a workspace virtualenv " + + "(python3 -m venv venv && venv/bin/pip install ) or, for a throwaway box, " + + "re-run with `pip install --break-system-packages `. Do not retry the identical command." + ); +}; + const matchers: ReadonlyArray = [ matchPythonModuleNotFound, matchVenvMissing, + matchExternallyManagedEnv, // Future: matchNodeModuleNotFound, matchCommandNotFound, matchEnvVarMissing, ... ]; diff --git a/packages/skills/src/tools/integrations/link/link-detector.test.ts b/packages/skills/src/tools/integrations/link/link-detector.test.ts index 8918f637cf..76573bcde0 100644 --- a/packages/skills/src/tools/integrations/link/link-detector.test.ts +++ b/packages/skills/src/tools/integrations/link/link-detector.test.ts @@ -112,3 +112,18 @@ describe("extractLinksFromMessage", () => { expect(result).toHaveLength(0); }); }); + +describe("URLs pasted inside JSON/config snippets", () => { + // Live: `"BASE_URL": "https://api.example.invalid/api/v2",` extracted the URL + // WITH its closing quote; the fetcher then requested `…/api/v2%22` and 404'd. + it("strips a trailing double quote from a JSON snippet", () => { + const urls = extractLinksFromMessage('Install: "BASE_URL": "https://api.example.invalid/api/v2",'); + expect(urls).toContain("https://api.example.invalid/api/v2"); + expect(urls.some((u) => u.includes('"'))).toBe(false); + }); + + it("strips trailing single quotes and braces from YAML/code shapes", () => { + expect(extractLinksFromMessage("url: 'https://example.invalid/x'")).toContain("https://example.invalid/x"); + expect(extractLinksFromMessage('{"u":"https://example.invalid/y"}')).toContain("https://example.invalid/y"); + }); +}); diff --git a/packages/skills/src/tools/integrations/link/link-detector.ts b/packages/skills/src/tools/integrations/link/link-detector.ts index 67a161f80b..d3269cac28 100644 --- a/packages/skills/src/tools/integrations/link/link-detector.ts +++ b/packages/skills/src/tools/integrations/link/link-detector.ts @@ -15,8 +15,11 @@ const MARKDOWN_LINK_RE = /\[[^\]]*\]\((https?:\/\/\S+?)\)/g; /** Regex to find bare URLs in text */ const BARE_URL_RE = /https?:\/\/\S+/g; -/** Characters that commonly trail URLs in natural text */ -const TRAILING_PUNCTUATION = new Set([".", ",", ")", "]", ">", ";", "!"]); +/** Characters that commonly trail URLs in natural text — including the quote + * family: a URL pasted inside a JSON/YAML/code snippet carries its closing + * quote into the `\S+` match (live: a config snippet yielded a fetch of + * `…/api/v2%22` — the encoded trailing `"` — which 404'd). */ +const TRAILING_PUNCTUATION = new Set([".", ",", ")", "]", ">", ";", "!", '"', "'", "`", "}"]); /** Hostnames considered localhost / loopback */ const LOCALHOST_NAMES = new Set(["localhost", "127.0.0.1", "::1", "[::1]"]); diff --git a/test/live/self-driving/scripts/.live-env.example b/test/live/self-driving/scripts/.live-env.example index 3f819fe8cd..cf4857b748 100644 --- a/test/live/self-driving/scripts/.live-env.example +++ b/test/live/self-driving/scripts/.live-env.example @@ -12,17 +12,25 @@ # them via _rig.mjs. For ad-hoc one-liners, `source .live-env` first. # # Only VPS is mandatory — every other value below IS the standard install.sh layout; override per box. +# +# EVERY value uses DEFAULT-ASSIGN (`: "${X:=...}"; export X`), never a bare `export X=...`, +# so a single run can target a DIFFERENT box without editing the file: +# +# VPS=comis-test1 ./install-vps.sh +# +# A bare `export` here silently CLOBBERS an inline override — which is exactly how a live-test +# run once ended up pointed at a PRODUCTION box. -export VPS="user@your-vps-host-or-ip" # ssh target — how you reach the test VPS (ssh KEY lives in ~/.ssh) -export REMOTE_SUDO="0" # 1 for an SSH deployment user with passwordless sudo -n -export COMIS_USER="comis" # the dedicated service user install.sh creates -export COMIS_HOME="/home/comis" # that user's home -export DATA="/home/comis/.comis" # daemon data dir (config.yaml, secrets.db, logs, workspace) -export PKG="/home/comis/.npm-global/lib/node_modules/comisai" # installed package root (daemon+cli dist live under its node_modules/@comis/) -export SERVICE="comis" # systemd unit name -export GW_PORT="4766" # gateway port (config.yaml gateway.port) -export CHATID="678314278" # emulator chat/sender id to drive -export EMU_DIR="/root/comis-emu" # emulator tree on the box (deploy-emu.sh rsyncs test/live/ here) +: "${VPS:=user@your-vps-host-or-ip}"; export VPS # ssh target — how you reach the test VPS (ssh KEY lives in ~/.ssh) +: "${REMOTE_SUDO:=0}"; export REMOTE_SUDO # 1 for an SSH deployment user with passwordless sudo -n +: "${COMIS_USER:=comis}"; export COMIS_USER # the dedicated service user install.sh creates +: "${COMIS_HOME:=/home/comis}"; export COMIS_HOME # that user's home +: "${DATA:=/home/comis/.comis}"; export DATA # daemon data dir (config.yaml, secrets.db, logs, workspace) +: "${PKG:=/home/comis/.npm-global/lib/node_modules/comisai}"; export PKG # installed package root (daemon+cli dist live under its node_modules/@comis/) +: "${SERVICE:=comis}"; export SERVICE # systemd unit name +: "${GW_PORT:=4766}"; export GW_PORT # gateway port (config.yaml gateway.port) +: "${CHATID:=678314278}"; export CHATID # emulator chat/sender id to drive +: "${EMU_DIR:=/root/comis-emu}"; export EMU_DIR # emulator tree on the box (deploy-emu.sh rsyncs test/live/ here) # Gateway bearer token for the RPC helpers (revoke.mjs & co). Under a production install `comis init` # stores it in the encrypted secrets store and config.yaml references ${COMIS_GATEWAY_TOKEN} — fetch @@ -30,4 +38,4 @@ export EMU_DIR="/root/comis-emu" # emulator tree on the box (deploy-emu.sh # ssh $VPS "su - comis -c 'comis secrets get COMIS_GATEWAY_TOKEN'" # (If your config.yaml carries a literal token instead, copy that. Fresh box with no config yet: # generate a fresh ≥32-char token and put the SAME literal in config.yaml gateway.tokens[].secret.) -export GWTOKEN="REPLACE-WITH-THE-BOX-GATEWAY-TOKEN-min-32-chars" +: "${GWTOKEN:=REPLACE-WITH-THE-BOX-GATEWAY-TOKEN-min-32-chars}"; export GWTOKEN From ae5329b5f4e4477db1fd1aed1dea3ee1162a8ba4 Mon Sep 17 00:00:00 2001 From: comis-agent Date: Sun, 26 Jul 2026 21:28:45 +0300 Subject: [PATCH 04/33] fix(executor): never narrate a turn whose reply was already delivered MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Empty-turn recovery exists to stop a SILENT turn — a tool batch that finished with no assistant text and told the user nothing. It fired on a turn that had already answered. Live shape: an onboarding turn sent its question with message({action:"send"}) rather than returning it as the final assistant text. The final message was therefore empty, recovery treated the turn as silent, and the user received a second bubble on top of the real answer: [comis: tool-call summary recovered from successful operations — the assistant's final message was empty] Completed 1 tool call in this batch: • message({action: "send"}) The work was done; the assistant did not summarize. Please ask "what did you do?" if details are needed. User actions: 9b1ba054 That is internal scaffolding — it names the tool protocol, instructs the user to interrogate the assistant, and trails a raw artifact id. Suppress the synthesis when the batch itself put words on the user's channel: a message send/reply carrying non-empty text. Everything that delivers NO words still synthesizes, so the silent-turn safety net is unchanged — a react, an attach with no text, and any non-message tool batch all keep the summary. The turn that took the tool path is left with an empty final response rather than a fabricated one; the INFO line records the suppression with the tool names so the underlying behaviour (a reply routed through the message tool instead of the response path) stays diagnosable. --- .../executor/executor-response-filter.test.ts | 53 ++++++++++++++++ .../src/executor/executor-response-filter.ts | 61 +++++++++++++++++++ 2 files changed, 114 insertions(+) diff --git a/packages/agent/src/executor/executor-response-filter.test.ts b/packages/agent/src/executor/executor-response-filter.test.ts index 6051e52580..ba51e6b1f8 100644 --- a/packages/agent/src/executor/executor-response-filter.test.ts +++ b/packages/agent/src/executor/executor-response-filter.test.ts @@ -886,3 +886,56 @@ describe("recoverEmptyFinalResponse — synthesis branch URL/code preservation", expect(result).not.toContain("https://api.example.com/data"); }); }); + +describe("empty-turn recovery does not narrate an already-delivered reply", () => { + // LIVE: an onboarding turn sent its question via message({action:"send"}), so the + // final assistant text was empty. Recovery then posted a SECOND bubble on top of + // the real answer: "[comis: tool-call summary recovered …] • message({action: + // "send"}) … Please ask what you did" plus a raw `User actions: `. The user + // read internal scaffolding. Recovery exists to stop a SILENT turn. + function recover(args: Record, toolName = "message"): string { + return recoverEmptyFinalResponse({ + extractedResponse: "", + // textEmitted:true is the live shape — text streamed during the turn but the + // FINAL extracted response was empty (the reply went out via the tool). + textEmitted: true, + messages: [ + { role: "user", content: [{ type: "text", text: "hi" }], timestamp: 1 }, + { + role: "assistant", + content: [{ type: "toolCall", id: "tc1", name: toolName, arguments: args }], + stopReason: "toolUse", + timestamp: 2, + }, + { role: "toolResult", toolCallId: "tc1", content: [{ type: "text", text: "OK" }], timestamp: 3 }, + { role: "assistant", content: [], stopReason: "stop", timestamp: 4 }, + ] as never, + logger: mockLogger(), + userMessageIndex: 0, + }); + } + + it("suppresses the synthesis when the batch already sent text to the channel", () => { + const out = recover({ action: "send", channel_type: "telegram", channel_id: "1", text: "the real reply" }); + expect(out).toBe(""); + expect(out).not.toContain("tool-call summary recovered"); + }); + + it("suppresses on a reply action too", () => { + expect(recover({ action: "reply", channel_type: "telegram", channel_id: "1", text: "answer", message_id: "9" })).toBe(""); + }); + + it("STILL synthesizes when the batch delivered NO words (a react is not a reply)", () => { + expect(recover({ action: "react", channel_type: "telegram", channel_id: "1", emoji: "\u{1F44D}" })) + .toContain("tool-call summary recovered"); + }); + + it("STILL synthesizes for a non-message tool batch (the silent-turn case it exists for)", () => { + expect(recover({ path: "a.txt" }, "write")).toContain("tool-call summary recovered"); + }); + + it("STILL synthesizes when a send carries no text (attach-only delivers no words)", () => { + expect(recover({ action: "send", channel_type: "telegram", channel_id: "1", text: " " })) + .toContain("tool-call summary recovered"); + }); +}); diff --git a/packages/agent/src/executor/executor-response-filter.ts b/packages/agent/src/executor/executor-response-filter.ts index 771dd8c005..be773380b0 100644 --- a/packages/agent/src/executor/executor-response-filter.ts +++ b/packages/agent/src/executor/executor-response-filter.ts @@ -211,6 +211,32 @@ export function recoverEmptyFinalResponse(params: { // Returning here is the ONE place that prevents the `standalone` walk-backward // (below) from ever firing alongside synthesis. Do not add code paths // that fall through to standalone after toolCallSummaries are non-empty. + // ALREADY-DELIVERED GUARD. When the batch itself put text on the user's + // channel (a `message` send/reply), the user has ALREADY read the + // assistant's words — the "empty" final message is not a silent turn, it + // is a turn whose reply took the tool path. Synthesizing here appends a + // SECOND, internal-looking bubble on top of the real answer: live, a user + // received the agent's onboarding question and then a + // "[comis: tool-call summary recovered …] • message({action: "send"}) … + // Please ask "what did you do?"" block plus a raw `User actions:` id. + // Recovery exists to stop a SILENT turn, not to narrate a delivered one. + if (deliveredUserFacingText(messages, lowerBound)) { + logger.info( + { + submodule: "executor.empty-turn-recovery", + recoveryPass: "suppressed-already-delivered", + toolCallCount: toolCallSummaries.length, + toolNames: [...toolNamesSet], + hint: + "Final assistant message was empty, but the tool batch already delivered text to " + + "the user's channel — suppressed the synthesized summary rather than posting " + + "internal recovery scaffolding on top of the real reply.", + }, + "Empty-turn recovery: suppressed (batch already delivered a reply)", + ); + return ""; + } + if (toolCallSummaries.length > 0) { const bullets = toolCallSummaries.map(s => ` • ${s}`).join("\n"); const synthesis = @@ -300,6 +326,41 @@ function extractVisibleText(content: any[]): string | undefined { * (internal mapped convention) for args. Returns bare tool name on malformed * input — never throws. */ /* eslint-disable @typescript-eslint/no-explicit-any */ +/** + * True when the assistant's tool batch ALREADY delivered user-facing text to the + * conversation's channel — i.e. the turn's reply took the `message` tool path + * rather than the final-assistant-text path. + * + * Only send/reply actions carrying non-empty `text` count. A `react`, `delete`, + * `fetch` or an attach-without-text delivers no words, so a turn that ends empty + * after one of those IS silent and still needs the synthesized summary. + * + * @param messages - the turn's message history. + * @param lowerBound - index of the first message in this turn's batch. + * @returns whether the user has already read the assistant's reply. + */ +function deliveredUserFacingText(messages: readonly any[], lowerBound: number): boolean { + const DELIVERING_ACTIONS = new Set(["send", "reply"]); + for (let i = lowerBound; i < messages.length; i++) { + const msg = messages[i]; // eslint-disable-line security/detect-object-injection + if (msg?.role !== "assistant" || !Array.isArray(msg.content)) continue; + for (const block of msg.content) { + if (block?.type !== "toolCall" && block?.type !== "tool_use") continue; + if (block?.name !== "message") continue; + const args: Record | undefined = + (block?.input && typeof block.input === "object" ? block.input : undefined) + ?? (block?.arguments && typeof block.arguments === "object" ? block.arguments : undefined); + if (!args) continue; + const action = typeof args.action === "string" ? args.action : undefined; + const text = typeof args.text === "string" ? args.text : ""; + if (action !== undefined && DELIVERING_ACTIONS.has(action) && text.trim().length > 0) { + return true; + } + } + } + return false; +} + function summarizeToolCall(call: any): string { const name = typeof call?.name === "string" ? call.name : "unknown_tool"; // Both Anthropic native (`input`) and internal mapped (`arguments`) shapes. From a33b8dc156617f53dd95ec68ea3001ec028ef2b5 Mon Sep 17 00:00:00 2001 From: comis-agent Date: Sun, 26 Jul 2026 22:13:11 +0300 Subject: [PATCH 05/33] fix(context): stop the assembler losing the turn's own request mid-tool-loop The LCD assembler concatenates store-backed history with a fresh tail sliced from the live array at a step-count boundary. Those two indices advance on different clocks: the store is written at afterTurn, so its horizon is frozen for the whole turn while the step boundary marches forward one step per LLM call. Once a turn ran more steps than freshTailTurns the boundary overtook the horizon, and every live message in between was assembled into neither segment. The first casualty is always the user message that STARTED the turn -- it sits exactly at the horizon, unpersisted until the turn ends. A long tool loop therefore dropped the request it was still serving, and the model answered as though it had never been asked, contradicting the user about what they wanted. Nothing reported it. The loss is not an eviction, so verdict stayed "fits", droppedCount stayed 0, and the window sat 91% idle. Ground truth from a live session: the store horizon held at 56 across a 7.5-minute turn while tailStart walked 41 -> 57; the final assembly shipped 72 of 73 live messages, and the one missing was the user's own request. Raising freshTailTurns cannot fix this -- it only moves which step opens the hole. Clamp the fresh-tail start to the persisted horizon, so freshTailTurns is a FLOOR on verbatim recency rather than a ceiling and anything the store cannot supply rides the verbatim tail instead. Gated on a complete read scope: countMessages answers 0 for an empty store, an incomplete (fail-closed) scope and a corrupt count alike, and clamping to 0 under fail-closed would make the whole live array unconditional and overflow a tight window. Clamping also stops the tail start sliding mid-turn, so the tail grows by pure suffix-append. Reconcile the concat seam against the live array on the INFO line that already rides every LLM call, netting out both deliberate removals (budget eviction and the residual trim) and measuring before transcript repair -- so a future regression here announces itself instead of needing a two-log-line hand-join. Separately, the MCP call-timeout hint closed by telling the caller to raise integrations.mcp.callToolTimeoutMs, which is an immutable config path. The agent followed the advice, the gateway rejected the patch, and the rejection surfaced to the user as a bare "[tool failure] gateway reported an error" stacked on top of an otherwise correct answer. The hint now names the knob for the operator and leaves the caller only the remedy it can actually apply. Also renames the fresh-tail-bound log's stepCount/stepSizes to tailSegmentCount/segmentSizes: that grouping opens a segment at every non-toolResult message, so it counts user messages too, unlike freshTailSteps which counts assistants. Reading one as the other inverts the diagnosis -- a segment count of steps+1 is the healthy shape, not an over-bound turn. --- docs/agents/compaction.mdx | 2 +- docs/get-started/glossary.mdx | 2 +- docs/reference/config-yaml.mdx | 2 +- docs/reference/json-rpc.mdx | 2 +- .../src/context-engine/lcd-assembler.test.ts | 64 +++++++++ .../agent/src/context-engine/lcd-assembler.ts | 109 +++++++++------ .../src/context-engine/lcd-coverage.test.ts | 128 ++++++++++++++++++ .../agent/src/context-engine/lcd-coverage.ts | 125 +++++++++++++++++ .../lcd-fresh-tail-bound.test.ts | 23 +++- .../context-engine/lcd-fresh-tail-bound.ts | 18 ++- .../mcp-client/mcp-client-call.test.ts | 26 ++++ .../mcp-client/mcp-client-call.ts | 5 +- 12 files changed, 451 insertions(+), 55 deletions(-) create mode 100644 packages/agent/src/context-engine/lcd-coverage.test.ts create mode 100644 packages/agent/src/context-engine/lcd-coverage.ts diff --git a/docs/agents/compaction.mdx b/docs/agents/compaction.mdx index 8f2d2b075c..bc5e24fc96 100644 --- a/docs/agents/compaction.mdx +++ b/docs/agents/compaction.mdx @@ -491,7 +491,7 @@ change anything. | `contextEngine.observationDeactivationChars` | number | `80000` | Character threshold to deactivate observation masking (20K-500K) | | `contextEngine.ephemeralKeepWindow` | number | `10` | Recent ephemeral tool results to preserve from masking (1-50) | | `contextEngine.contextThreshold` | number | `0.75` | **DAG mode:** context-utilization fraction (of the turn's effective budget window — the reconciled context window under any capability-class cap) that triggers a leaf-summarization pass at the end of a turn (0.1-0.95) | -| `contextEngine.freshTailTurns` | number | `8` | **DAG mode:** most-recent steps (assistant + tool round-trips) always kept verbatim and never summarized/evicted (1-50). Reduced only on a small context window (see [config reference](/reference/config-yaml)); honoured as configured on frontier-sized windows. **Raise it for agents that run long tool loops** — a turn with more tool round-trips than this slides the user's own request out of the verbatim tail, and the model can then answer as though it was never asked. | +| `contextEngine.freshTailTurns` | number | `8` | **DAG mode:** most-recent steps (assistant + tool round-trips) always kept verbatim and never summarized/evicted (1-50). Reduced only on a small context window (see [config reference](/reference/config-yaml)); honoured as configured on frontier-sized windows. This is a **floor**, not a ceiling: the tail is additionally widened to cover everything the store has not yet persisted, so a turn that runs more tool round-trips than this cannot slide its own originating request out of context. Steps older than the tail are still recoverable — they resolve from the store as reconstructed history rather than verbatim blocks. | | `contextEngine.leafChunkTokens` | number | `20000` | **DAG mode:** token cap for the oldest out-of-tail chunk summarized into one leaf summary (1K-100K); clamped at runtime to the resolved summarizer model's window — the smaller of its configured window and the probed served window when the summarizer runs on the served-bound provider — minus the summary target, template overhead, and previous-summary size, so a small compaction summarizer or a served-bound primary is never fed an over-window chunk; a single message larger than the clamped cap is replaced by a bounded deterministic extraction (no LLM call) | | `contextEngine.leafTargetTokens` | number | `1200` | **DAG mode:** target token size for a leaf summary (96-5000) | | `contextEngine.deferCompaction` | boolean | `true` | **DAG mode:** run the afterTurn leaf + condense passes in the background on the per-conversation serializer (never blocking the turn). `false` runs them inline (deterministic, for tests) | diff --git a/docs/get-started/glossary.mdx b/docs/get-started/glossary.mdx index 4201955418..f948790b81 100644 --- a/docs/get-started/glossary.mdx +++ b/docs/get-started/glossary.mdx @@ -132,7 +132,7 @@ When an agent uses a tool, it may need another thinking step to process the tool ## Fresh Tail -The most recent **steps** in a DAG-mode conversation that are always kept verbatim (an assistant message plus the tool results it triggered counts as one step — not user-turns), configured via `contextEngine.freshTailTurns` (default 8; reduced only on a small context window). The tail is bounded by **step count**, not tokens — so a turn with more tool round-trips than this can push the originating user request out of verbatim context even when the context window is nearly empty. `comis explain` reports the effective and configured values as `contextBudget.freshTailSteps` / `freshTailStepsConfigured`. The fresh tail is sliced as the original structured blocks (never reconstructed-from-text) and is never evicted. This is a DAG context engine concept; DAG is the default engine (`contextEngine.version` defaults to `"dag"`; set `"pipeline"` to opt into the simpler engine). See [Context Management](/agents/context-management#dag-mode-the-default-engine). +The most recent **steps** in a DAG-mode conversation that are always kept verbatim (an assistant message plus the tool results it triggered counts as one step — not user-turns), configured via `contextEngine.freshTailTurns` (default 8; reduced only on a small context window). The step count is a **floor**: the tail additionally always reaches back at least to the store's persisted horizon, because the store is written at turn end and anything newer exists only in the live message array. That guarantees the request that started the current turn stays verbatim no matter how many tool round-trips the turn runs. `comis explain` reports the effective and configured values as `contextBudget.freshTailSteps` / `freshTailStepsConfigured`. The fresh tail is sliced as the original structured blocks (never reconstructed-from-text) and is never evicted. This is a DAG context engine concept; DAG is the default engine (`contextEngine.version` defaults to `"dag"`; set `"pipeline"` to opt into the simpler engine). See [Context Management](/agents/context-management#dag-mode-the-default-engine). ## Gateway diff --git a/docs/reference/config-yaml.mdx b/docs/reference/config-yaml.mdx index 35f11d9d30..a245bcfb47 100644 --- a/docs/reference/config-yaml.mdx +++ b/docs/reference/config-yaml.mdx @@ -1879,7 +1879,7 @@ External service integrations for search, MCP servers, media processing, and aut | Key | Type | Default | Description | |-----|------|---------|-------------| -| `callToolTimeoutMs` | `number` | `120000` | Default timeout for MCP tool calls in ms. On expiry the tool error **names this key and its current value** and states that an identical retry re-expires the same deadline — narrow the request (smaller page / date window / fewer entities) or raise this value. Slow report and media tools may need well over two minutes. | +| `callToolTimeoutMs` | `number` | `120000` | Default timeout for MCP tool calls in ms. On expiry the tool error **names this key and its current value** and states that an identical retry re-expires the same deadline, so the caller narrows the request (smaller page / date window / fewer entities) instead. This key is an **immutable config path** — an agent cannot patch it via the `gateway` tool, so the error says so explicitly and directs the change to an operator. Raising it means editing this file and restarting the daemon. Slow report and media tools may need well over two minutes. | | `servers` | `McpServerEntry[]` | `[]` | List of MCP servers | **MCP Server Entry (integrations.mcp.servers[])** diff --git a/docs/reference/json-rpc.mdx b/docs/reference/json-rpc.mdx index fc90707fa9..af8ce7c9b5 100644 --- a/docs/reference/json-rpc.mdx +++ b/docs/reference/json-rpc.mdx @@ -784,7 +784,7 @@ durably reset a task transition, the higher-priority reconciliation authority and the `health_signal:background_task_recovery_failed` system-health signal. -For `context_exhausted` sessions the report carries an optional **`contextBudget`** section -- the terminal per-LLM-call budget equation extracted from the trajectory's `context.budget` records (last record wins): `{ windowTokens, rawContextWindowTokens, windowCapSource, systemTokens, freshTailTokens, budgetedHistoryTokens, keptCount, assembledInputTokens, outputHeadroom, verdict, freshTailSteps?, freshTailStepsConfigured? }`. `freshTailSteps` / `freshTailStepsConfigured` are the **effective vs operator-configured** count of trailing conversation steps kept verbatim (`contextEngine.freshTailTurns`). The verbatim tail is bounded by **step count, not tokens**, so a turn with more tool round-trips than the effective bound slides the user's own request out of verbatim context *while `verdict` still reads `fits`* and the window sits nearly empty. When the effective value is below the configured one, `likelyRootCause` stamps a `fresh_tail_clamped` verdict naming both numbers and the knob. `windowCapSource` names what clamped the effective window below the model's configured `contextWindow`: the exact `contextEngine.budget.*` knob (`effectiveContextCapSmall` / `effectiveContextCapNano`), `served` (the Ollama-served `num_ctx` bound the window -- fix with `OLLAMA_CONTEXT_LENGTH= ollama serve` or a Modelfile `PARAMETER num_ctx`), `capabilityClass` (the operator's `providers.entries..capabilities.capabilityClass` pin bound the window -- pin a higher class or remove the pin; the `contextEngine.budget.*` caps do not move this bind), or `none`. `verdict` is the fit-check outcome (`fits` / `downshifted` / `exhausted`). When present, `likelyRootCause` is numbers-backed: it states the assembled-vs-window totals, the binding cap, the system+tools share of the window, and the kept-history count, and its suggested steps name the binding lever (the budget knob to raise, the Ollama serve knobs, or the `capabilityClass` pin) or recommend reducing the active tool surface when tool schemas dominate. The section is absent when the trajectory carries no `context.budget` records; the verdict then falls back to the generic wording. +For `context_exhausted` sessions the report carries an optional **`contextBudget`** section -- the terminal per-LLM-call budget equation extracted from the trajectory's `context.budget` records (last record wins): `{ windowTokens, rawContextWindowTokens, windowCapSource, systemTokens, freshTailTokens, budgetedHistoryTokens, keptCount, assembledInputTokens, outputHeadroom, verdict, freshTailSteps?, freshTailStepsConfigured? }`. `freshTailSteps` / `freshTailStepsConfigured` are the **effective vs operator-configured** count of trailing conversation steps kept verbatim (`contextEngine.freshTailTurns`). The verbatim tail is bounded by **step count, not tokens**, but that count is a floor: the tail also always reaches back to the store's persisted horizon, so the current turn's own request cannot fall out of context however many tool round-trips the turn runs. When the effective value is below the configured one, `likelyRootCause` stamps a `fresh_tail_clamped` verdict naming both numbers and the knob. `windowCapSource` names what clamped the effective window below the model's configured `contextWindow`: the exact `contextEngine.budget.*` knob (`effectiveContextCapSmall` / `effectiveContextCapNano`), `served` (the Ollama-served `num_ctx` bound the window -- fix with `OLLAMA_CONTEXT_LENGTH= ollama serve` or a Modelfile `PARAMETER num_ctx`), `capabilityClass` (the operator's `providers.entries..capabilities.capabilityClass` pin bound the window -- pin a higher class or remove the pin; the `contextEngine.budget.*` caps do not move this bind), or `none`. `verdict` is the fit-check outcome (`fits` / `downshifted` / `exhausted`). When present, `likelyRootCause` is numbers-backed: it states the assembled-vs-window totals, the binding cap, the system+tools share of the window, and the kept-history count, and its suggested steps name the binding lever (the budget knob to raise, the Ollama serve knobs, or the `capabilityClass` pin) or recommend reducing the active tool surface when tool schemas dominate. The section is absent when the trajectory carries no `context.budget` records; the verdict then falls back to the generic wording. When the session ran any memory recalls the report carries an optional **`recall`** section -- the memory-recall outcome aggregated over the trajectory's `memory.recalled` records: `{ recalls, zeroHits, lastLanes, lastFinalCount, rerankerAvailable, crossUserRecalls?, lastCrossUserCount? }` (counts and booleans only -- never the query text or any recalled memory body). `recalls` is how many recalls ran, `zeroHits` how many returned **no** injected memories (a recall miss), and the `last*` fields describe the terminal recall (its lane count and final injected-memory count). **`crossUserRecalls`** (present only when `> 0`) is how many recalls injected at least one memory scoped to a **different user** than the conversation -- the cross-sender privacy signal for agent-scoped recall (one shared agent surfacing sender A's memory into sender B's turn), so "was cross-sender data injected into this turn's context?" is answerable from the always-on report without pre-enabling the opt-in `diagnostics.recallTrace` artifact; **`lastCrossUserCount`** is the terminal recall's cross-user injected count. Counts only -- never the user-ids or bodies. This drives a dedicated **`recall_miss`** `likelyRootCause` verdict: a degraded session whose recalls **all** missed (`zeroHits === recalls`) and that matched no tool/context/breaker cause is root-caused as a recall miss, pointing the operator at the recall **scope** (agent- vs user-scoped) and, for non-Latin queries, the trigram-twin lanes (see [`comis system-health`](#obs-system-health) `health_signal` / `config_posture` embedder). A zero-hit recall on a **healthy** turn is benign -- the agent simply did not need memory -- and never names a cause. The section is absent for sessions with no recall records. diff --git a/packages/agent/src/context-engine/lcd-assembler.test.ts b/packages/agent/src/context-engine/lcd-assembler.test.ts index a9c39bfc96..057d266e28 100644 --- a/packages/agent/src/context-engine/lcd-assembler.test.ts +++ b/packages/agent/src/context-engine/lcd-assembler.test.ts @@ -409,6 +409,70 @@ describe("createLcdContextEngine", () => { expect(out[11]).toBe(live[11]); }); + it("a LONG tool loop never opens a hole between the store horizon and the fresh tail", async () => { + // The coverage invariant: `history ∪ freshTail == the live array`. History can + // only ever reach the store's persisted horizon, and the store is written at + // afterTurn — so mid-turn the horizon is FROZEN while the fresh tail's + // step-count boundary marches FORWARD one step per LLM call. Once a turn runs + // more steps than `freshTailTurns`, the boundary passes the horizon and the + // messages in between belong to NEITHER segment: not yet in the store, and + // older than the verbatim tail. The first casualty is always the user message + // that STARTED the turn (it sits at the horizon, unpersisted until the turn + // ends), so the model loses the request it is mid-way through serving and + // contradicts the user about what they asked for. + // + // Persisted horizon = 9 completed messages; the live array then carries the + // turn's own request + a 3-step tool loop that outruns freshTailTurns=2. + const persistedPrefix: Message[] = [ + userMsg("u0"), + assistantText("a0"), + userMsg("u1"), + assistantText("a1"), + userMsg("u2"), + assistantText("a2"), + userMsg("u3"), + assistantText("a3"), + userMsg("u4"), + ]; + for (let i = 0; i < persistedPrefix.length; i++) append(store, persistedPrefix[i] as Message, i); + + const live: AgentMessage[] = [ + ...(persistedPrefix as AgentMessage[]), + // index 9 — the request that started this turn. Unpersisted (the turn is + // still running) and, once the loop is 3 steps deep, older than the tail. + userMsg("THE-REQUEST") as AgentMessage, + assistantToolCall("tu_1", "read", {}) as AgentMessage, // 10 + toolResult("tu_1", "read", "r1") as AgentMessage, // 11 + assistantToolCall("tu_2", "read", {}) as AgentMessage, // 12 + toolResult("tu_2", "read", "r2") as AgentMessage, // 13 + assistantToolCall("tu_3", "read", {}) as AgentMessage, // 14 + toolResult("tu_3", "read", "r3") as AgentMessage, // 15 + ]; + + const { deps } = makeDeps(store); + // freshTailTurns=2 → the step boundary lands on tu_2 (index 12), which is + // PAST the 9-message store horizon. Live[9..11] — the request and the first + // tool step — are covered by neither segment. + const engine = createLcdContextEngine(dagConfig(2), deps); + const out = await engine.transformContext(live); + + const texts = out.map((m) => { + const c = (m as unknown as { content: unknown }).content; + if (typeof c === "string") return c; + const arr = c as { type: string; text?: string }[]; + return arr.find((b) => b.type === "text")?.text ?? ""; + }); + // The turn's own request must survive — exactly once. + expect(texts.filter((t) => t === "THE-REQUEST")).toEqual(["THE-REQUEST"]); + // And nothing else in the gap is silently dropped: every tool step is present. + const callIds = out.flatMap((m) => + roleOf(m) === "assistant" + ? ((m as unknown as { content: unknown[] }).content.filter(isToolCallBlock).map((b) => b.id)) + : [], + ); + expect(callIds).toEqual(["tu_1", "tu_2", "tu_3"]); + }); + it("live array SHRINKS below the store count — assembler over-includes nothing, doubles nothing", async () => { // A future heal/compaction could reassign state.messages SMALLER than the // append-only store. The assembler seam must stay robust to live.length <= diff --git a/packages/agent/src/context-engine/lcd-assembler.ts b/packages/agent/src/context-engine/lcd-assembler.ts index ae71696dbd..90cd06fe04 100644 --- a/packages/agent/src/context-engine/lcd-assembler.ts +++ b/packages/agent/src/context-engine/lcd-assembler.ts @@ -71,6 +71,7 @@ import type { Message } from "@earendil-works/pi-ai"; import { estimateMessageTokens } from "../safety/token-estimator.js"; import { sanitizeToolUseResultPairing } from "./transcript-repair.js"; import { resolveClampedFreshTailTurns } from "../model/fresh-tail-clamp.js"; +import { resolveFreshTailStart, warnOnCoverageShortfall } from "./lcd-coverage.js"; import { computeTokenBudgetForProfile } from "./budget-capacity-cap.js"; import { FAIL_CLOSED_PROFILE } from "../executor/model-profile.js"; import { runPreflightFitCheck } from "./lcd-preflight.js"; @@ -224,20 +225,32 @@ export function createLcdContextEngine( // Clamp freshTailTurns to what the effective window can afford. // deps.modelProfile?.contextWindow is Infinity for frontier/mid — clamp never fires. const effectiveWindow = deps.modelProfile?.contextWindow ?? Infinity; - // The PROTECTED fresh tail ships UNCONDITIONALLY (the eviction cannot - // trim it), so on a tight window it is bounded to the RESIDUAL room - // (boundFreshTailTotalToResidual, below). The residual (= window − S − floor headroom - // − preamble) is computed JUST BEFORE the bound — AFTER `profile` is resolved — so it - // reads `profile.reasoningStyle`, the EXACT value the pre-flight uses (a - // pre-resolution `deps.modelProfile` read could diverge from the pre-flight's `profile` - // and under-count the native reasoning reserve → overflow). The clamp here keeps only - // its original 30%-of-window upper bound (no residual input — a turn-count estimate is - // skewed by a single oversized message, which the per-message bound handles anyway). + // The PROTECTED fresh tail ships UNCONDITIONALLY (the eviction cannot trim it), + // so on a tight window it is bounded to the RESIDUAL room + // (boundFreshTailTotalToResidual, below). That residual (= window − S − floor + // headroom − preamble) is computed JUST BEFORE the bound — AFTER `profile` is + // resolved — so it reads `profile.reasoningStyle`, the EXACT value the pre-flight + // uses (a pre-resolution `deps.modelProfile` read could diverge and under-count + // the native reasoning reserve → overflow). The clamp here keeps only its 30%-of- + // window upper bound (no residual input — a turn-count estimate is skewed by a + // single oversized message, which the per-message bound handles anyway). const clampedFreshTailTurns = resolveClampedFreshTailTurns( effectiveWindow, config.freshTailTurns, ); - const tailStart = freshTailBoundaryIndex(liveMessages, clampedFreshTailTurns); + // persistedMsgCount is the TOTAL count of persisted messages in scope — NOT + // `rows.length`, the BOUNDED working set (message-refs only), which undercounts + // once the oldest messages fold into summary-refs and corrupts the fresh-tail/ + // eviction overlap below. `countMessages` is a bounded COUNT(*) — one integer, NO + // O(total-history) row fetch. Fail-closed: no read scope ⇒ 0. + const persistedMsgCount = store.countMessages(readScope); + // COVERAGE: the step boundary and the store horizon come from different clocks, + // and a turn deeper than `freshTailTurns` drives the boundary past the horizon + // — opening a hole that silently swallows the turn's own originating request. + // lcd-coverage.ts owns the invariant + why the clamp needs a readable scope. + const scopeIsReadable = deps.agentId !== undefined && deps.tenantId !== undefined; + const stepBoundary = freshTailBoundaryIndex(liveMessages, clampedFreshTailTurns); + const tailStart = resolveFreshTailStart(stepBoundary, persistedMsgCount, scopeIsReadable); // The fresh tail is sliced VERBATIM from the live array (it bypasses the // store, so the ingest-time neutralization in executor/lcd-ingest.ts does not // reach it). Neutralize any forged context-boundary markers an assistant turn @@ -255,6 +268,12 @@ export function createLcdContextEngine( configuredFreshTailSteps: config.freshTailTurns, freshTailCount: rawFreshTail.length, tailStart, + // Coverage seam, one-read visible: `inFlightGapCovered > 0` means the tail + // was WIDENED past its step bound to carry unpersisted messages — the turn + // shape that used to lose the user's own request. + stepBoundary, + persistedMsgCount, + inFlightGapCovered: Math.max(0, stepBoundary - tailStart), agentId: deps.agentId, sessionKey: deps.sessionKey, }, @@ -282,37 +301,30 @@ export function createLcdContextEngine( // those map to RAW message-refs at the END of `context_items` (summaries // collapse the OLDEST run, so the tail of the view is raw). // - // Count/length robustness: `rawOverlap` is a RAW-message count (`persistedMsgCount`, - // the bounded COUNT(*) total — NOT `rows.length`, which is only the referenced - // working-set subset), while the slice indexes into the COLLAPSED - // `resolved` view (`resolved.length ≤ persistedMsgCount` once any leaf/condense - // pass has run). - // Subtracting `rawOverlap` from `resolved.length` directly is correct ONLY - // under the oldest-run-collapse invariant; if the fresh-tail window reaches - // back further than the trailing raw run (a large `freshTailTurns`, or a - // future non-oldest collapse), `resolved.length − rawOverlap` would slice - // ACROSS the head summary-ref and silently DROP the oldest history. So we - // bound the exclusion by `trailingMessageRefs` — the count of message-refs - // at the END of `resolved`, stopping at the first summary-ref — so the - // evictable-prefix slice can NEVER cut into a summary-ref. Under the normal - // invariant `rawOverlap ≤ trailingMessageRefs`, so this is byte-identical - // to the prior behavior; when the invariant is stressed it degrades to a - // benign double at the seam (transcript repair re-pairs it) rather than a - // silent drop. + // Count/length robustness: `rawOverlap` is a RAW-message count + // (`persistedMsgCount`, the bounded COUNT(*) total), while the slice indexes + // into the COLLAPSED `resolved` view (`resolved.length ≤ persistedMsgCount` + // once any leaf/condense pass has run). Subtracting `rawOverlap` from + // `resolved.length` directly is correct ONLY under the oldest-run-collapse + // invariant; if the fresh-tail window reaches back further than the trailing + // raw run (a large `freshTailTurns`, or a future non-oldest collapse), it + // would slice ACROSS the head summary-ref and silently DROP the oldest + // history. So we bound the exclusion by `trailingMessageRefs` — the count of + // message-refs at the END of `resolved`, stopping at the first summary-ref — + // so the evictable-prefix slice can NEVER cut into a summary-ref. Under the + // normal invariant `rawOverlap ≤ trailingMessageRefs`, so this is byte- + // identical to the prior behavior; when the invariant is stressed it degrades + // to a benign double at the seam (transcript repair re-pairs it), not a drop. // // Drop-free + double-free for BOTH L>H (mid-turn: the store lags the live // array by the in-flight delta, so the in-flight tail rides only via - // `freshTail`) and L<=H (a heal shrank the live array). - // persistedMsgCount is the TOTAL count of persisted - // messages in scope — NOT `rows.length`. `rows` is the BOUNDED working - // set (message-refs only); once the oldest messages fold into summary-refs, - // `rows.length` undercounts the total and corrupts the fresh-tail/eviction - // overlap below (symptom: a broken - // fresh tail + a mis-placed condensed summary). `countMessages` is a bounded - // COUNT(*) — one integer, NO O(total-history) row fetch — so assembly is - // byte-identical to a full `getMessages(readScope).length` read while the - // row fetch stays O(referenced-ids). Fail-closed: no read scope ⇒ 0. - const persistedMsgCount = store.countMessages(readScope); + // `freshTail` — which holds ONLY because `tailStart` is clamped to the + // horizon above; without that clamp the delta beyond the fresh tail's step + // bound rode NOTHING) and L<=H (a heal shrank the live array). + // `tailStart` is clamped to `persistedMsgCount` above, so this subtraction + // can no longer go negative — the max(0,…) is a retained guard, not a live + // branch (a negative here WAS the silent-hole bug: it clamped to "no + // overlap" and let the gap fall out of both segments). const rawOverlap = Math.max(0, persistedMsgCount - tailStart); let trailingMessageRefs = 0; for (let i = resolvedKinds.length - 1; i >= 0 && resolvedKinds[i] === "message"; i--) { @@ -522,6 +534,22 @@ export function createLcdContextEngine( // The fresh tail is concatenated UNCONDITIONALLY — never evicted. const assembled = [...budgeted, ...freshTail]; + // Reconcile THIS seam against the live conversation — the check that would have + // made the silent hole self-reporting (lcd-coverage.ts owns the why). + const coverageShortfall = warnOnCoverageShortfall(deps.logger, { + liveCount: liveMessages.length, + assembledCount: assembled.length, + droppedCount, + freshTailTrimmedCount: rawFreshTail.length - freshTail.length, + persistedMsgCount, + stepBoundary, + tailStart, + historyCount: budgeted.length, + freshTailCount: freshTail.length, + agentId: deps.agentId, + sessionKey: deps.sessionKey, + }); + // 5. NORMALIZE assistant string content to array blocks. const normalized = assembled.map(normalizeAssistantContent); @@ -566,6 +594,11 @@ export function createLcdContextEngine( historyCount: budgeted.length, freshTailCount: freshTail.length, assembledCount: repaired.length, + // The coverage reconciliation, inline: `liveCount` is what the + // assembled array must account for, `coverageShortfall` is what it + // failed to (0 on every healthy call). + liveCount: liveMessages.length, + coverageShortfall, // The budget equation at INFO: diagnosability must not depend on // logLevel having been debug before an incident // (the lcd-evict budget fields are DEBUG). One line per LLM call. diff --git a/packages/agent/src/context-engine/lcd-coverage.test.ts b/packages/agent/src/context-engine/lcd-coverage.test.ts new file mode 100644 index 0000000000..7780d7d0d0 --- /dev/null +++ b/packages/agent/src/context-engine/lcd-coverage.test.ts @@ -0,0 +1,128 @@ +// SPDX-License-Identifier: Apache-2.0 +/** + * Tests for the assembler's coverage seam — `history ∪ freshTail == liveMessages`. + * + * The end-to-end proof lives in lcd-assembler.test.ts ("a LONG tool loop never + * opens a hole…"); these pin the two helpers' edge semantics directly, because + * both of them fail SILENTLY when wrong: a mis-clamped tail start drops messages + * with every reported number still reading healthy, and a mis-computed shortfall + * either cries wolf on every turn (and gets ignored) or never fires at all. + */ +import { describe, it, expect } from "vitest"; +import { resolveFreshTailStart, warnOnCoverageShortfall } from "./lcd-coverage.js"; +import { createMockLogger } from "../../../../test/support/mock-logger.js"; + +describe("resolveFreshTailStart", () => { + it("clamps the step boundary back to the store horizon (the in-flight gap)", () => { + // The production shape: 9 steps deep into a turn, the boundary (12) has + // overtaken the frozen horizon (9). Everything from 9 on is unpersisted and + // must ride the verbatim tail. + expect(resolveFreshTailStart(12, 9, true)).toBe(9); + }); + + it("leaves the boundary alone when it is still inside the persisted range", () => { + // The steady state — the tail is a pure step-count window and nothing widens. + expect(resolveFreshTailStart(41, 56, true)).toBe(41); + }); + + it("is a no-op at the exact seam", () => { + expect(resolveFreshTailStart(56, 56, true)).toBe(56); + }); + + it("does NOT clamp under an unreadable scope — 0 there means 'unknown', not 'empty'", () => { + // `countMessages` fail-closes to 0 for an incomplete scope. Clamping to 0 + // would make the WHOLE live array unconditional and overflow a tight window, + // instead of degrading honestly to the fresh tail. + expect(resolveFreshTailStart(17, 0, false)).toBe(17); + }); + + it("clamps to 0 on a genuinely empty store — a first turn is entirely in flight", () => { + expect(resolveFreshTailStart(3, 0, true)).toBe(0); + }); +}); + +describe("warnOnCoverageShortfall", () => { + const base = { + persistedMsgCount: 56, + stepBoundary: 57, + tailStart: 56, + historyCount: 56, + freshTailCount: 17, + freshTailTrimmedCount: 0, + agentId: "agent_a", + sessionKey: "sess-a", + }; + + it("stays silent when the assembled array covers the live conversation", () => { + const logger = createMockLogger(); + const shortfall = warnOnCoverageShortfall(logger as never, { + ...base, + liveCount: 73, + assembledCount: 73, + droppedCount: 0, + }); + expect(shortfall).toBe(0); + expect(logger.warn).not.toHaveBeenCalled(); + }); + + it("nets out deliberate eviction — dropped history is accounted for, not a defect", () => { + const logger = createMockLogger(); + const shortfall = warnOnCoverageShortfall(logger as never, { + ...base, + liveCount: 73, + assembledCount: 60, + droppedCount: 13, + }); + expect(shortfall).toBe(0); + expect(logger.warn).not.toHaveBeenCalled(); + }); + + it("nets out the residual trim — a tight window dropping oldest tail steps is honest, not a hole", () => { + const logger = createMockLogger(); + const shortfall = warnOnCoverageShortfall(logger as never, { + ...base, + liveCount: 73, + assembledCount: 69, + droppedCount: 0, + freshTailTrimmedCount: 4, + }); + expect(shortfall).toBe(0); + expect(logger.warn).not.toHaveBeenCalled(); + }); + + it("never reports a NEGATIVE shortfall (a larger assembled array is not a defect)", () => { + const logger = createMockLogger(); + const shortfall = warnOnCoverageShortfall(logger as never, { + ...base, + liveCount: 73, + assembledCount: 75, + droppedCount: 0, + }); + expect(shortfall).toBe(0); + expect(logger.warn).not.toHaveBeenCalled(); + }); + + it("WARNs with the two indices that localize the gap when messages go missing", () => { + const logger = createMockLogger(); + // The live incident: boundary 57 past horizon 56 → live[56] in neither segment. + const shortfall = warnOnCoverageShortfall(logger as never, { + ...base, + tailStart: 57, // un-clamped — the pre-fix behaviour + liveCount: 73, + assembledCount: 72, + droppedCount: 0, + }); + expect(shortfall).toBe(1); + expect(logger.warn).toHaveBeenCalledTimes(1); + const fields = logger.warn.mock.calls[0]![0] as Record; + expect(fields.errorKind).toBe("internal"); + expect(fields.coverageShortfall).toBe(1); + // Both sides of the seam ride the line, so the gap is localizable without a + // second lens — that hand-join across two log lines is what made this bug + // take a full investigation to find. + expect(fields.persistedMsgCount).toBe(56); + expect(fields.stepBoundary).toBe(57); + expect(String(fields.hint)).toContain("persistedMsgCount"); + expect(String(fields.hint)).toContain("stepBoundary"); + }); +}); diff --git a/packages/agent/src/context-engine/lcd-coverage.ts b/packages/agent/src/context-engine/lcd-coverage.ts new file mode 100644 index 0000000000..1942fe3a8b --- /dev/null +++ b/packages/agent/src/context-engine/lcd-coverage.ts @@ -0,0 +1,125 @@ +// SPDX-License-Identifier: Apache-2.0 +/** + * The LCD `dag` assembler's COVERAGE seam: `history ∪ freshTail == liveMessages`. + * + * History is reconstructed from the STORE, so it can never reach past the store's + * persisted horizon; the fresh tail is sliced from the LIVE array at a boundary + * derived from a STEP COUNT. Those two indices come from different clocks. The + * store is written at `afterTurn`, so mid-turn the horizon is FROZEN while the + * step boundary marches FORWARD one step per LLM call — and once a turn runs more + * steps than `freshTailTurns`, the boundary overtakes the horizon and the messages + * between them belong to NEITHER segment. + * + * The first casualty is always the user message that STARTED the turn: it sits + * exactly at the horizon, unpersisted until the turn ends. A long tool loop then + * silently drops the request it is still serving, and the model answers as though + * it was never asked — while every number the engine reports says healthy + * (`verdict:"fits"`, `droppedCount:0`, a nearly-idle window), because the message + * was never ASSEMBLED rather than evicted. + * + * This module owns both halves of the fix: {@link resolveFreshTailStart} closes + * the gap by construction, and {@link warnOnCoverageShortfall} makes any future + * regression in the seam announce itself on the line that already rides every + * LLM call. + * + * Extracted from lcd-assembler.ts to keep that file under the 800-line cap. + * + * @module + */ + +import type { ComisLogger } from "@comis/core"; + +/** + * The fresh tail's slice-start index, clamped so the verbatim tail always reaches + * back to at least the store's persisted horizon. + * + * `freshTailTurns` stays a FLOOR on verbatim recency (never fewer than N steps) — + * this only ever WIDENS the tail, and only by content that exists nowhere else. + * It also makes the mid-turn tail start STABLE (the horizon does not move until + * `afterTurn`), so the tail grows by pure suffix-append instead of sliding its + * prefix on every step — which is strictly kinder to the provider prompt cache. + * + * Gated on a COMPLETE read scope. `countMessages` answers 0 for three different + * states — an empty store, an incomplete (fail-closed) scope, and a corrupt count + * — and only the first means "these messages are in flight". Under the fail-closed + * scope history is empty for an unrelated reason (cross-agent leak protection), so + * clamping to 0 there would make the WHOLE live array unconditional and overflow a + * tight window instead of degrading honestly to the fresh tail. + * + * @param stepBoundary - the step-count boundary from `freshTailBoundaryIndex`. + * @param persistedMsgCount - the store's horizon (`countMessages`, 0 when unreadable). + * @param scopeIsReadable - whether agentId+tenantId are both present (the store's own guard). + * @returns the slice-start index for the fresh tail. + */ +export function resolveFreshTailStart( + stepBoundary: number, + persistedMsgCount: number, + scopeIsReadable: boolean, +): number { + return scopeIsReadable ? Math.min(stepBoundary, persistedMsgCount) : stepBoundary; +} + +/** + * Reconcile the assembled array against the live conversation and WARN when it + * does not cover it. + * + * Diagnosing the original hole took hand-joining `historyCount + freshTailCount` + * against the NEXT call's `messageCount` across two log lines, because no single + * line carried both sides. This reconciles them at the source. + * + * Measured at the CONCAT seam (`budgeted ++ freshTail`), NOT on the repaired + * output, and with every DELIBERATE removal netted out — otherwise the WARN cries + * wolf on healthy turns and gets ignored, which is worse than not having it: + * - `droppedCount` — budget eviction of the history prefix; + * - `freshTailTrimmedCount` — the residual trim dropping oldest tail steps on a + * tight window (honest degradation, and the pre-flight reports it); + * - transcript repair, excluded by measuring before it runs. Repair both ADDS + * synthesized results for unpaired calls and DROPS orphan/duplicate ones, so + * its output is not a coverage measure of this seam at all. + * + * @returns the shortfall (0 on every healthy call), for the caller's INFO line. + */ +export function warnOnCoverageShortfall( + logger: ComisLogger, + m: { + liveCount: number; + assembledCount: number; + droppedCount: number; + freshTailTrimmedCount: number; + persistedMsgCount: number; + stepBoundary: number; + tailStart: number; + historyCount: number; + freshTailCount: number; + agentId: string | undefined; + sessionKey: string | undefined; + }, +): number { + const shortfall = Math.max( + 0, + m.liveCount - m.droppedCount - m.freshTailTrimmedCount - m.assembledCount, + ); + if (shortfall === 0) return 0; + logger.warn( + { + step: "lcd-assemble", + errorKind: "internal" as const, + liveCount: m.liveCount, + persistedMsgCount: m.persistedMsgCount, + stepBoundary: m.stepBoundary, + tailStart: m.tailStart, + historyCount: m.historyCount, + freshTailCount: m.freshTailCount, + assembledCount: m.assembledCount, + droppedCount: m.droppedCount, + freshTailTrimmedCount: m.freshTailTrimmedCount, + coverageShortfall: shortfall, + hint: + "assembled fewer messages than the live array minus deliberate eviction — history and the fresh tail no longer cover the conversation between them. Compare persistedMsgCount (the store horizon) against stepBoundary: when the boundary is larger, live messages in that range belong to neither segment and are silently missing from the prompt.", + agentId: m.agentId, + sessionKey: m.sessionKey, + }, + "lcd assembled context does NOT cover the live conversation", + ); + return shortfall; +} diff --git a/packages/agent/src/context-engine/lcd-fresh-tail-bound.test.ts b/packages/agent/src/context-engine/lcd-fresh-tail-bound.test.ts index dfc0caabcf..ca58effbe9 100644 --- a/packages/agent/src/context-engine/lcd-fresh-tail-bound.test.ts +++ b/packages/agent/src/context-engine/lcd-fresh-tail-bound.test.ts @@ -191,16 +191,29 @@ describe("boundProtectedFreshTail — instrumentation + the live OpenAI growth r // Each step is small, so the trim CAN reduce the tail to ≤ residual (+ one kept step). const oneStep = factoredMessageTokens(tail[0]!) + factoredMessageTokens(tail[1]!); expect(outFactored).toBeLessThanOrEqual(residual + oneStep); - // Instrumentation present: stepCount = 28 (14×2 non-toolResult msgs), dropped > 0. + // Instrumentation present: tailSegmentCount = 28 (14×2 non-toolResult msgs), dropped > 0. + // Named `tailSegment*`, NOT `step*`: this grouping opens a segment at every + // non-toolResult message, so USER messages count too — unlike `freshTailSteps` + // on the lcd-fresh-tail line, which counts assistants only. Reading one as the + // other inverts the diagnosis (a segment count of steps+1 is the healthy shape, + // not an over-bound turn), so the two names must stay distinct. const call = logger.debug.mock.calls.find((c) => (c[0] as { step?: string }).step === "fresh-tail-bound") ?? logger.warn.mock.calls.find((c) => (c[0] as { step?: string }).step === "fresh-tail-bound"); expect(call, "fresh-tail-bound log must be emitted").toBeDefined(); - const p = call![0] as { stepCount: number; stepSizes: number[]; droppedSteps: number; freshTailResidual: number }; - expect(p.stepCount).toBe(28); - expect(p.stepSizes.length).toBe(28); // NOT one giant step — grouping splits the turns - expect(p.droppedSteps).toBeGreaterThan(0); // it actually trimmed + const p = call![0] as { + tailSegmentCount: number; + segmentSizes: number[]; + droppedSegments: number; + freshTailResidual: number; + }; + expect(p.tailSegmentCount).toBe(28); + expect(p.segmentSizes.length).toBe(28); // NOT one giant step — grouping splits the turns + expect(p.droppedSegments).toBeGreaterThan(0); // it actually trimmed expect(p.freshTailResidual).toBe(residual); + // The ambiguous names are gone — a log reader cannot resurrect the misread. + expect(p).not.toHaveProperty("stepCount"); + expect(p).not.toHaveProperty("stepSizes"); }); it("WARN fires when the protected tail CANNOT be trimmed below the residual (single oversized current turn)", () => { diff --git a/packages/agent/src/context-engine/lcd-fresh-tail-bound.ts b/packages/agent/src/context-engine/lcd-fresh-tail-bound.ts index 550aeda050..3f4c60a581 100644 --- a/packages/agent/src/context-engine/lcd-fresh-tail-bound.ts +++ b/packages/agent/src/context-engine/lcd-fresh-tail-bound.ts @@ -311,7 +311,7 @@ export function boundProtectedFreshTail( // protected tail is/isn't trimmed — not a hypothesis. Per-step factored sizes + // kept/dropped counts disambiguate the 4 candidate causes (giant step / wrong residual // / preamble / keep-last-too-much). `roleOf` mirrors the function's own grouping so the - // logged stepCount IS what the trim saw. + // logged tailSegmentCount IS what the trim saw. const roleAt = (m: AgentMessage): string | undefined => (m as { role?: string }).role; const stepStartIdx: number[] = []; for (let i = 0; i < freshTail.length; i++) { @@ -345,10 +345,16 @@ export function boundProtectedFreshTail( systemTokens: ctx.systemTokens, freshTailPreambleTokens: ctx.freshTailPreambleTokens, effectiveWindow: ctx.effectiveWindow, - stepCount: stepStartIdx.length, - stepSizes, // each step's factored tokens — exposes a giant un-droppable step - keptSteps, - droppedSteps: stepStartIdx.length - keptSteps, + // NOT the same "step" as `freshTailSteps` on the lcd-fresh-tail line: that + // one counts ASSISTANT messages, this grouping counts every non-toolResult + // message, so a user message opens a segment too. Reading a + // `tailSegmentCount` of 9 as "9 assistant steps, over the 8-step bound" + // reverses the diagnosis — it is 8 steps plus the user message that opened + // them, which is the HEALTHY shape. Named apart so the two cannot be confused. + tailSegmentCount: stepStartIdx.length, + segmentSizes: stepSizes, // each segment's factored tokens — exposes a giant un-droppable step + keptSegments: keptSteps, + droppedSegments: stepStartIdx.length - keptSteps, freshTailFactoredBefore: before, freshTailFactoredAfter: after, // the bounded total — should be ≤ residual when trimmable fitsResidual, @@ -363,7 +369,7 @@ export function boundProtectedFreshTail( { ...logPayload, errorKind: "resource" as const, - hint: "protected fresh tail could NOT be trimmed below the residual — the kept (current) step(s) exceed it; the pre-flight will exhaust. Check stepSizes for a single giant step (grouping) vs a genuinely oversized current turn (oversized_input).", + hint: "protected fresh tail could NOT be trimmed below the residual — the kept (current) step(s) exceed it; the pre-flight will exhaust. Check segmentSizes for a single giant step (grouping) vs a genuinely oversized current turn (oversized_input).", }, "lcd protected fresh tail STILL exceeds the residual after trimming", ); diff --git a/packages/skills/src/skills/integrations/mcp-client/mcp-client-call.test.ts b/packages/skills/src/skills/integrations/mcp-client/mcp-client-call.test.ts index 1a268ab9ee..5f59ea0c35 100644 --- a/packages/skills/src/skills/integrations/mcp-client/mcp-client-call.test.ts +++ b/packages/skills/src/skills/integrations/mcp-client/mcp-client-call.test.ts @@ -219,6 +219,32 @@ describe("call timeout names its knob", () => { expect(result.error.message).toMatch(/narrow/i); }); + it("does not send the agent to patch the knob — that config path is immutable at runtime", async () => { + // Observed live: the hint's closing clause read "or raise + // `integrations.mcp.callToolTimeoutMs` for this deployment", so the agent did + // exactly that — one `gateway` patch call, rejected by the immutable-path guard + // ("Cannot patch immutable config path: integrations.mcp.callToolTimeoutMs. + // Patchable: integrations.mcp.servers."). The rejection surfaced in the chat as a + // bare "[tool failure] gateway reported an error" on top of the real answer. + // Naming the knob is right (an operator needs it); telling the AGENT to raise it + // is not — the only remedy available to the caller is narrowing the request. + const serverName = "vendor-mcp"; + const state = makeConnectedState(serverName, () => + Promise.reject(new McpError(ErrorCode.RequestTimeout, "Request timed out")), + ); + + const result = await callTool(state, deps, `mcp:${serverName}/vendor_activity_report`, {}); + if (result.ok) throw new Error("expected err"); + const message = result.error.message; + + // Still names the knob and its value — the operator-facing half is unchanged. + expect(message).toContain("integrations.mcp.callToolTimeoutMs"); + // But never phrases it as an action the caller can take. + expect(message).not.toMatch(/\braise\b|\bincrease\b/i); + // …and says whose job it is, so the agent stops instead of trying a patch. + expect(message).toMatch(/operator/i); + }); + it("emits a WARN carrying the same hint + a dependency errorKind", async () => { const serverName = "vendor-mcp"; const state = makeConnectedState(serverName, () => diff --git a/packages/skills/src/skills/integrations/mcp-client/mcp-client-call.ts b/packages/skills/src/skills/integrations/mcp-client/mcp-client-call.ts index 08e83d2fe3..2f63070e08 100644 --- a/packages/skills/src/skills/integrations/mcp-client/mcp-client-call.ts +++ b/packages/skills/src/skills/integrations/mcp-client/mcp-client-call.ts @@ -405,7 +405,8 @@ export function mcpCallTimeoutHint( `MCP tool "${toolName}" on server "${serverName}" timed out — it exceeded the call ` + `deadline of ${timeoutMs}ms (\`integrations.mcp.callToolTimeoutMs\`, currently ${timeoutMs}). ` + "This deadline is deterministic — do not retry it unchanged, the same call re-expires it. " + - "Either narrow the request (a smaller page/date window/fewer entities) so it completes " + - `inside ${timeoutMs}ms, or raise \`integrations.mcp.callToolTimeoutMs\` for this deployment.` + `Narrow the request (a smaller page/date window/fewer entities) so it completes inside ${timeoutMs}ms. ` + + "The deadline itself cannot be changed from here — it is an immutable config path, so only an " + + "operator can adjust it by editing the config file and restarting the daemon." ); } From fe8bc41b9c1d27aaec5e102dcf514cb909216a12 Mon Sep 17 00:00:00 2001 From: comis-agent Date: Sun, 26 Jul 2026 22:46:35 +0300 Subject: [PATCH 06/33] fix(session): keep the approval handle when scrubbing a redacted secret call An approval-gated secret write returns `requiresConfirmation: true` with an opaque `pending_action_id`, and instructs the model to re-call with that id and `_confirmed: true` while OMITTING the value -- the value is replayed server-side precisely because it is about to be redacted out of the model's context. That handle is the whole mechanism for completing a gated write without asking the user to send the credential twice. scrubRedactedToolCalls threw it away. The scrub keys on the tool_use ARGS carrying the redaction placeholder, which is correct -- the args hold the secret. But it then also replaced the matching tool_RESULT, which holds no secret at all, with an opaque "(prior secret operation -- no output shown)", and summarised the assistant turn as "The action completed; do not retry" for a call that had never run. Live consequence, both halves visible in one session: the user pasted MCP credentials, three env_set calls each returned a pending_action_id, the user replied "yes" -- and the agent had no handle to confirm with and believed the secrets were already stored. It listed secrets, found zero, and asked the user to resend the username and password in cleartext over the chat channel. That is the exact deadlock the pending_action_id was introduced to break, defeated one layer below where it was implemented. Index the gated results by toolCallId before the scrub runs, then carry ONLY the opaque id through: the result replacement names the id and the confirm shape, and the assistant summary states the action is pending confirmation rather than complete. Nothing else from the result survives -- the id is extracted, never the surrounding payload -- so the secret still never re-enters replay. Ungated redacted calls keep their existing text byte-for-byte. --- .../session/scrub-redacted-tool-calls.test.ts | 86 ++++++++++++++ .../src/session/scrub-redacted-tool-calls.ts | 108 +++++++++++++++++- 2 files changed, 192 insertions(+), 2 deletions(-) diff --git a/packages/agent/src/session/scrub-redacted-tool-calls.test.ts b/packages/agent/src/session/scrub-redacted-tool-calls.test.ts index 09445d7bfd..299db773e9 100644 --- a/packages/agent/src/session/scrub-redacted-tool-calls.test.ts +++ b/packages/agent/src/session/scrub-redacted-tool-calls.test.ts @@ -272,3 +272,89 @@ describe("scrubRedactedToolCalls", () => { expect(rewriteCalls).toBe(0); }); }); + +describe("scrubRedactedToolCalls — approval-gated secrets keep their recovery handle", () => { + // Observed live. The user pasted MCP credentials; the agent called env_set three + // times; each returned requiresConfirmation with a `pending_action_id` and a hint + // saying to re-call with that id and _confirmed:true, OMITTING the value (the + // value is replayed server-side precisely because it gets redacted from context). + // The user replied "yes" — and the agent had nothing to confirm with, because this + // scrub had replaced the RESULT (which held the id) with an opaque placeholder and + // told the model the action "completed". It listed secrets, found none, and asked + // the user to re-paste the password in cleartext on Telegram: the exact deadlock + // the pending_action_id exists to break. + // + // The tool_use ARGS hold the secret and must still be scrubbed. The RESULT holds + // no secret — only an opaque server-side handle — so the handle must survive. + const gatedResultText = JSON.stringify({ + requiresConfirmation: true, + actionType: "env.set", + pending_action_id: "c7a047d5-4023-462d-a9b6-03568c4edbb1", + hint: "NOT performed — present it to the user; re-call with _confirmed: true.", + }); + + function gatedEntries() { + return [ + msg("user", [{ type: "text", text: "store these" }]), + msg("assistant", [ + { + type: "toolCall", + id: "tc_1", + name: "gateway", + arguments: { action: "env_set", env_key: "VENDOR_API_PASSWORD", env_value: "[REDACTED]" }, + }, + ]), + toolResult("tc_1", gatedResultText), + ]; + } + + it("preserves the pending_action_id so the confirm can still be issued", () => { + const fileEntries = gatedEntries(); + const result = scrubRedactedToolCalls({ fileEntries } as any); + expect(result.scrubbed).toBe(true); + + const replaced = (fileEntries[2] as any).message.content[0].text as string; + expect(replaced).toContain("c7a047d5-4023-462d-a9b6-03568c4edbb1"); + expect(replaced).toMatch(/_confirmed/); + }); + + it("still strips the secret VALUE from the replayed tool_use args", () => { + const fileEntries = gatedEntries(); + scrubRedactedToolCalls({ fileEntries } as any); + const blob = JSON.stringify(fileEntries); + expect(blob).not.toContain("env_value"); + // The key name is fine to keep; the value never reappears. + expect(blob).toContain("VENDOR_API_PASSWORD"); + }); + + it("does NOT tell the model a gated action completed — it never ran", () => { + const fileEntries = gatedEntries(); + scrubRedactedToolCalls({ fileEntries } as any); + const summary = (fileEntries[1] as any).message.content[0].text as string; + expect(summary).not.toMatch(/completed/i); + expect(summary).not.toMatch(/do not retry/i); + // …and it must say what IS still required. + expect(summary).toMatch(/confirm/i); + }); + + it("an UNGATED redacted call still reports completion (no behaviour change)", () => { + const fileEntries = [ + msg("user", [{ type: "text", text: "store it" }]), + msg("assistant", [ + { + type: "toolCall", + id: "tc_9", + name: "gateway", + arguments: { action: "env_set", env_key: "PLAIN_KEY", env_value: "[REDACTED]" }, + }, + ]), + toolResult("tc_9", JSON.stringify({ ok: true })), + ]; + scrubRedactedToolCalls({ fileEntries } as any); + const summary = (fileEntries[1] as any).message.content[0].text as string; + expect(summary).toMatch(/completed/i); + expect((fileEntries[2] as any).message.content[0].text).toBe( + "(prior secret operation — no output shown)", + ); + }); +}); diff --git a/packages/agent/src/session/scrub-redacted-tool-calls.ts b/packages/agent/src/session/scrub-redacted-tool-calls.ts index 773d1f630d..1fe0e120d4 100644 --- a/packages/agent/src/session/scrub-redacted-tool-calls.ts +++ b/packages/agent/src/session/scrub-redacted-tool-calls.ts @@ -32,6 +32,7 @@ import type { SessionManager } from "@earendil-works/pi-coding-agent"; import { REDACTED_TOOL_RESULT_USER_MESSAGE } from "./synthetic-user-messages.js"; +import { tryCatch } from "@comis/shared"; import { getSessionFileEntries } from "./session-manager-internals.js"; /** Literal placeholder written by sanitizeSessionSecrets. */ @@ -76,6 +77,28 @@ export function scrubRedactedToolCalls( // Assistant entry indices marked for full content replacement. const fullyPoisonedAssistants = new Map(); // idx -> summary + // Pass 0: index the APPROVAL-GATED results by toolCallId. + // + // A gated call returns `requiresConfirmation: true` plus an opaque + // `pending_action_id`; the secret value stays server-side and the model is told + // to re-call with that id and `_confirmed: true`, OMITTING the value — precisely + // because the value is about to be redacted out of its context. Scrubbing that + // RESULT away destroys the only handle that can complete the action, and the + // model's sole remaining move is to ask the user to re-paste the secret in + // cleartext. The tool_use ARGS carry the secret and must still be scrubbed; the + // result carries no secret, so the handle survives (nothing else does — the id + // is extracted, never the surrounding payload). + const pendingActionIdByToolCallId = new Map(); + for (const entry of fileEntries) { + if (!entry || entry.type !== "message") continue; + const m = entry.message; + if (!m || (m.role !== "toolResult" && m.role !== "tool")) continue; + const id = typeof m.toolCallId === "string" ? m.toolCallId : undefined; + if (id === undefined) continue; + const pendingId = extractPendingActionId(m.content); + if (pendingId !== undefined) pendingActionIdByToolCallId.set(id, pendingId); + } + for (let idx = 0; idx < fileEntries.length; idx++) { const entry = fileEntries[idx]; if (!entry || entry.type !== "message") continue; @@ -103,7 +126,13 @@ export function scrubRedactedToolCalls( const toolCallId = typeof block.id === "string" ? block.id : undefined; const toolName = typeof block.name === "string" ? block.name : "tool"; - const summary = buildSummaryText(toolName, args); + const summary = buildSummaryText( + toolName, + args, + toolCallId === undefined + ? undefined + : pendingActionIdByToolCallId.get(toolCallId), + ); if (toolCallId) candidateIds.push(toolCallId); poisonedInThisMessage++; @@ -153,9 +182,15 @@ export function scrubRedactedToolCalls( if (!toolCallId || !poisoned.has(toolCallId)) continue; if (msg.role !== "toolResult" && msg.role !== "tool") continue; + const pendingId = pendingActionIdByToolCallId.get(toolCallId); msg.role = "user"; msg.content = [ - { type: "text", text: REDACTED_TOOL_RESULT_USER_MESSAGE }, + { + type: "text", + text: pendingId === undefined + ? REDACTED_TOOL_RESULT_USER_MESSAGE + : pendingApprovalReplayText(pendingId), + }, ]; delete msg.toolCallId; delete msg.toolName; @@ -198,7 +233,24 @@ function argsContainPlaceholder(args: Record): boolean { function buildSummaryText( toolName: string, args: Record, + pendingActionId?: string, ): string { + // A gated call did NOT run. Telling the model it "completed" while its result + // is scrubbed away is how the live deadlock started: the model believed the + // secrets were stored, listed them, found nothing, and asked the user to paste + // the password again. State the truth and name the handle that finishes it. + if (pendingActionId !== undefined) { + const key = + typeof args.env_key === "string" ? args.env_key : "the secret"; + return ( + `(Awaiting your confirmation to set secret ${key} — the call was gated ` + + `and has NOT run. Its arguments are elided from replay. Once the user ` + + `approves, re-call the SAME action with pending_action_id: ` + + `"${pendingActionId}" and _confirmed: true, OMITTING the value — it is ` + + `replayed server-side. Never ask the user to resend the secret, and ` + + `never pass a placeholder like [REDACTED].)` + ); + } if (toolName === "gateway" && args.action === "env_set") { const key = typeof args.env_key === "string" ? args.env_key : "the secret"; @@ -216,3 +268,55 @@ function buildSummaryText( `reuse a [REDACTED] placeholder.)` ); } + +/** + * The user-role replacement for a scrubbed APPROVAL-GATED tool result. + * + * Carries ONLY the opaque `pending_action_id` — never the surrounding payload — + * so the confirm remains issuable while the secret stays out of replay. Without + * it the model loses its only route to completing the action and falls back to + * asking the user to resend the credential in cleartext. + */ +function pendingApprovalReplayText(pendingActionId: string): string { + return ( + `(prior secret operation — output withheld. It is still PENDING your ` + + `confirmation. To complete it, re-call the same action with ` + + `pending_action_id: "${pendingActionId}" and _confirmed: true, omitting ` + + `the value — it is replayed server-side. Do not ask for the secret again.)` + ); +} + +/** + * Pull an approval gate's `pending_action_id` out of a tool result's content. + * + * Gated results are JSON text blocks. Parse when possible; fall back to a + * bounded regex so a wrapper (offload notice, security banner) around the JSON + * still yields the handle. Returns undefined for every non-gated result, which + * keeps the ungated scrub path byte-identical. + */ +function extractPendingActionId(content: unknown): string | undefined { + const text = collectText(content); + if (text === undefined || !text.includes("pending_action_id")) return undefined; + const parsed = tryCatch(() => JSON.parse(text) as unknown); + if (parsed.ok) { + const value = (parsed.value as { pending_action_id?: unknown } | null) + ?.pending_action_id; + if (typeof value === "string" && value.length > 0) return value; + } + const match = /"pending_action_id"\s*:\s*"([^"]{1,200})"/.exec(text); + return match?.[1]; +} + +/** Flatten a tool result's content blocks (or string shorthand) to text. */ +function collectText(content: unknown): string | undefined { + if (typeof content === "string") return content; + if (!Array.isArray(content)) return undefined; + const parts: string[] = []; + for (const block of content) { + if (block && typeof block === "object") { + const t = (block as { text?: unknown }).text; + if (typeof t === "string") parts.push(t); + } + } + return parts.length > 0 ? parts.join("\n") : undefined; +} From 1069afed4403f172e0a977adc085c1d909476743 Mon Sep 17 00:00:00 2001 From: comis-agent Date: Sun, 26 Jul 2026 22:46:48 +0300 Subject: [PATCH 07/33] fix(locale): stop a client's UI language enforcing over the conversation The response-locale resolver has three tiers: an operator pin (`agents..language`), a channel-supplied request locale, and a script inferred from the current message. `docs/operations/multilingual.mdx` already specifies the contract -- "the two inferred tiers ... are advisory: they inform the prompt but never trigger a repair", and only the operator pin enforces. The code did not match its own spec. The request tier was built with enforceLocale: true, which made a DEVICE SETTING the strongest signal in the system -- stronger than the language the conversation was actually being held in, whose script-derived tier is advisory by design. Live: a Hebrew conversation, correctly answered in Hebrew from the advisory und-Hebr tier. The next message was an English technical instruction, which does not contradict a Telegram client language_code of "en", so the transport tier engaged, outranked the conversation, and the agent switched to English and stayed there. The resolver's own comment already called this tier "TRANSPORT metadata (a client UI language_code, a caller-supplied field) -- a device setting, not the conversation's language"; it just was not treated that way. Mark the request tier advisory, matching the script tier beside it and the documented contract. An operator who needs a guaranteed reply language sets `agents..language`, which still enforces and still outranks the device locale. The script-contradiction guard is unchanged. Updates the five assertions that encoded the old enforcing behaviour, and makes the config reference state plainly that only the explicit key enforces -- it previously read as though any resolved locale could trigger a repair turn. --- docs/reference/config-yaml.mdx | 2 +- .../src/executor/prompt-assembly.test.ts | 5 +- .../resolve-response-locale-policy.test.ts | 53 +++++++++++++++++-- .../resolve-response-locale-policy.ts | 14 ++++- 4 files changed, 65 insertions(+), 9 deletions(-) diff --git a/docs/reference/config-yaml.mdx b/docs/reference/config-yaml.mdx index a245bcfb47..5fe8b781e7 100644 --- a/docs/reference/config-yaml.mdx +++ b/docs/reference/config-yaml.mdx @@ -257,7 +257,7 @@ Multi-agent configuration map. Each key is an agent ID, and the value is a `PerA | `preserveRecent` | `number` | `4` | Minimum recent messages to always preserve during compaction | | `workspacePath` | `string` | _(unset)_ | Path to agent workspace directory containing identity files | | `reactionLevel` | `enum` | _(unset)_ | Reaction frequency: `minimal` (1 per 5-10 exchanges) or `extensive` (react freely) | -| `language` | `string` | _(unset)_ | Explicit response locale, consumed by the deterministic degraded replies (context-exhausted / output-starved notices) and locale-quality observability. Must be a canonical BCP-47 tag (`"he"`, `"pt-BR"`) — validation rejects anything else. When omitted, Comis resolves the turn's locale from the channel-supplied request locale (e.g. the Telegram client's `language_code` — ignored for the turn when the message's script contradicts it), then the dominant non-Latin script of the current message (as a script-only locale such as `und-Hebr`), else leaves it unset. A resolved locale also governs the live reply: a script-mismatched final response triggers at most one bounded tools-disabled repair turn (see [Multilingual](/operations/multilingual)). | +| `language` | `string` | _(unset)_ | Explicit response locale, consumed by the deterministic degraded replies (context-exhausted / output-starved notices) and locale-quality observability. Must be a canonical BCP-47 tag (`"he"`, `"pt-BR"`) — validation rejects anything else. When omitted, Comis resolves the turn's locale from the channel-supplied request locale (e.g. the Telegram client's `language_code` — ignored for the turn when the message's script contradicts it), then the dominant non-Latin script of the current message (as a script-only locale such as `und-Hebr`), else leaves it unset. **Only this explicit key ENFORCES the reply language**: when it is set, a script-mismatched final response triggers at most one bounded tools-disabled repair turn. Both inferred tiers are advisory — they inform the prompt but never trigger a repair, so a client's UI language cannot override the language the conversation is actually being held in. Set this key when you need a fixed reply language regardless of any individual message (see [Multilingual](/operations/multilingual)). | | `enforceFinalTag` | `boolean` | `false` | When enabled, only content inside `` blocks reaches users. Suppresses all content outside final tags on both streaming and non-streaming paths. | | `fastMode` | `boolean` | `false` | Enables fast mode for the LLM provider (provider-specific behavior). | | `storeCompletions` | `boolean` | `false` | When enabled, sends `store: true` to OpenAI-compatible providers for completion storage. | diff --git a/packages/agent/src/executor/prompt-assembly.test.ts b/packages/agent/src/executor/prompt-assembly.test.ts index ce7832aa5a..bbc01c6084 100644 --- a/packages/agent/src/executor/prompt-assembly.test.ts +++ b/packages/agent/src/executor/prompt-assembly.test.ts @@ -3100,7 +3100,8 @@ describe("bootstrap file snapshotting", () => { expect(result.responseLocalePolicy).toEqual({ locale: "fr-CA", source: "request", - enforceLocale: true, + // Transport tier is advisory — only an operator pin enforces. + enforceLocale: false, }); }); @@ -4267,7 +4268,7 @@ describe("parent prefix reuse", () => { expect(result.responseLocalePolicy).toEqual({ locale: "ar-EG", source: "request", - enforceLocale: true, + enforceLocale: false, }); }); diff --git a/packages/agent/src/executor/resolve-response-locale-policy.test.ts b/packages/agent/src/executor/resolve-response-locale-policy.test.ts index 49a9b973a3..f5e70d767f 100644 --- a/packages/agent/src/executor/resolve-response-locale-policy.test.ts +++ b/packages/agent/src/executor/resolve-response-locale-policy.test.ts @@ -23,7 +23,8 @@ describe("resolveResponseLocalePolicy", () => { locale: "fr-CA", source: "request", translationTarget: "ja-JP", - enforceLocale: true, + // Transport tier: resolved and offered as a hint, but never enforced. + enforceLocale: false, }); }); @@ -56,7 +57,7 @@ describe("resolveResponseLocalePolicy", () => { confidence: "high", }); expect(resolveLocale({ requestLocale: "fr-CA" })).toEqual({ - policy: { locale: "fr-CA", source: "request", enforceLocale: true }, + policy: { locale: "fr-CA", source: "request", enforceLocale: false }, confidence: "medium", }); expect(resolveLocale({})).toEqual({ @@ -165,7 +166,9 @@ describe("request-locale vs conversation-script precedence", () => { }); expect(policy.locale).toBe("he"); expect(policy.source).toBe("request"); - expect(policy.enforceLocale).toBe(true); + // Agreeing with the message script makes the hint more likely RIGHT, not + // more authoritative — it is still a device setting. + expect(policy.enforceLocale).toBe(false); }); it("prefers the current Latin prose script over a contradicting non-Latin transport locale (advisory)", () => { @@ -185,7 +188,7 @@ describe("request-locale vs conversation-script precedence", () => { })).toEqual({ locale: "he", source: "request", - enforceLocale: true, + enforceLocale: false, }); }); @@ -195,7 +198,7 @@ describe("request-locale vs conversation-script precedence", () => { requestText: "", }); expect(policy.locale).toBe("he"); - expect(policy.enforceLocale).toBe(true); + expect(policy.enforceLocale).toBe(false); }); it("never overrides the explicit operator locale with the message script", () => { @@ -239,3 +242,43 @@ describe("an operator pin is the ONLY enforcer of response locale", () => { }); }); }); + +describe("transport-tier locale is advisory, never enforced", () => { + // Observed live: a Hebrew conversation ("שלום" → Hebrew reply, correctly, from + // the ADVISORY und-Hebr script tier). The user's next message was an English + // technical instruction, which does not contradict their Telegram client's + // language_code of "en" — so the transport tier took over and, because it was + // marked enforceLocale:true, outranked the conversation's own signal. The agent + // switched to English mid-conversation. + // + // The asymmetry was the bug: a DEVICE SETTING enforced, while the conversation's + // actual language was only ever advisory. This module already documents the + // request tier as "TRANSPORT metadata … a device setting, not the conversation's + // language" — so it must not be the strongest signal in the system. Only an + // operator pin (`explicitLocale`, source "explicit") enforces. + it("does not enforce a client UI language over the conversation", () => { + const policy = resolveResponseLocalePolicy({ + requestLocale: "en", + requestText: "Install this MCP:\nnpx -y some-mcp", + }); + expect(policy.source).toBe("request"); + expect(policy.enforceLocale).toBe(false); + }); + + it("an operator pin still enforces and still outranks the device locale", () => { + expect(resolveResponseLocalePolicy({ + explicitLocale: "he", + requestLocale: "en", + requestText: "Install this MCP", + })).toEqual({ locale: "he", source: "explicit", enforceLocale: true }); + }); + + it("a script-contradicting message still drops the device locale entirely", () => { + // Unchanged behaviour: Hebrew text under an "en" device locale falls through + // to the advisory script tier rather than being repaired toward English. + expect(resolveResponseLocalePolicy({ + requestLocale: "en", + requestText: "שלום, מה שלומך היום?", + })).toEqual({ locale: "und-Hebr", source: "request", enforceLocale: false }); + }); +}); diff --git a/packages/agent/src/executor/resolve-response-locale-policy.ts b/packages/agent/src/executor/resolve-response-locale-policy.ts index 6769b50991..9e743b44b6 100644 --- a/packages/agent/src/executor/resolve-response-locale-policy.ts +++ b/packages/agent/src/executor/resolve-response-locale-policy.ts @@ -41,7 +41,19 @@ export function resolveResponseLocalePolicy( ): ResponseLocalePolicy { const candidates: ReadonlyArray = [ [input.explicitLocale, "explicit", true], - [input.requestLocale, "request", true], + // ADVISORY, never ENFORCED — for the same reason the script tier below is. + // This tier is a DEVICE SETTING (a client UI language_code), so making it + // enforcing left the strongest signal in the system belonging to the one + // input that is not about the conversation at all. Live: a Hebrew + // conversation, one English technical instruction — which does not + // contradict a client language_code of "en", so the transport tier engaged + // and, being enforcing, outranked the conversation's own (advisory) Hebrew + // signal. The agent switched to English mid-conversation and stayed there. + // + // Only an OPERATOR PIN (`explicitLocale`) enforces. An operator who needs a + // guaranteed response language sets `agents..language`; everything else + // is a hint, which is all a per-message or per-device signal can honestly be. + [input.requestLocale, "request", false], ]; const translationTarget = canonicalLocale(input.translationTarget); for (const [raw, source, enforceLocale] of candidates) { From d01f48927ea98e9d4ce84301e175dfb2b7391328 Mon Sep 17 00:00:00 2001 From: comis-agent Date: Sun, 26 Jul 2026 23:31:24 +0300 Subject: [PATCH 08/33] fix(safety): count a background task's failure against the tool that failed The per-tool retry breaker is recorded against the tool that RETURNED the error. Auto-backgrounding invalidates that assumption: it converts a slow tool into a non-error "moved to the background" result, so the originating tool reports SUCCESS on every launch and its consecutive-failure counter resets each time. It can never trip. The real failure surfaces later on the `background_tasks` POLLER, which relays it -- and the poller trips instead. The effect is the exact inverse of what a breaker is for: the agent loses its ability to OBSERVE outcomes while the tool that is actually failing stays completely unthrottled. Live: a heavy report tool was launched 20+ times between 19:45:02 and 19:53:54, each burning the full 120s MCP call deadline, against a 600s execution budget. The breaker did fire -- `tool.breaker_opened {toolName:"background_tasks", consecutiveFailures:2, errorTag:"conflict"}` -- but on the poller, so nothing slowed the launches. The turn died at 599s on `execution.aborted {reason:"pipeline_timeout"}` and the user got a canned error after ten minutes of waiting. Both halves are needed; either alone makes it worse. Stop blaming the poller for relaying a failure it merely reported (its OWN failures -- a bad taskId, a storage error -- still count, so this is not a blanket amnesty), and count the failure against the originating tool. `background_task:failed` already carries that tool name, stamped by the manager from the task record, so nothing has to be parsed out of the poller's prose error text, which names the tool only in free-form English. The subscription's lifetime matches the per-execution breaker it feeds and is torn down in the same finally as `unsubscribeCacheTrace`, so no listener outlives the execution that owns it. --- packages/agent/src/bridge/pi-event-bridge.ts | 15 ++- .../src/executor/pi-executor/pi-executor.ts | 17 +++ .../background-failure-attribution.test.ts | 119 ++++++++++++++++++ .../safety/background-failure-attribution.ts | 118 +++++++++++++++++ 4 files changed, 268 insertions(+), 1 deletion(-) create mode 100644 packages/agent/src/safety/background-failure-attribution.test.ts create mode 100644 packages/agent/src/safety/background-failure-attribution.ts diff --git a/packages/agent/src/bridge/pi-event-bridge.ts b/packages/agent/src/bridge/pi-event-bridge.ts index 9643739ef9..1829fa06d0 100644 --- a/packages/agent/src/bridge/pi-event-bridge.ts +++ b/packages/agent/src/bridge/pi-event-bridge.ts @@ -11,6 +11,7 @@ */ import { shouldCompact } from "@earendil-works/pi-coding-agent"; +import { isRelayedBackgroundFailure } from "../safety/background-failure-attribution.js"; import type { AgentSessionEvent } from "@earendil-works/pi-coding-agent"; import type { AssistantMessage } from "@earendil-works/pi-ai"; import { @@ -1248,7 +1249,19 @@ export function createPiEventBridge(deps: PiEventBridgeDeps): PiEventBridgeResul // Runtime guards do not describe tool health. Feeding them into the // per-tool retry breaker would blame a healthy tool for a local // execution budget and manufacture a provider-outage diagnosis. - if (deps.toolRetryBreaker && classifiedFailureBy !== "runtime_guard") { + // The poller relaying SOMEONE ELSE's failure is the poller working + // correctly. Counting it blames the one tool the agent needs in order to + // observe outcomes, while the tool that actually failed reports success + // on every launch (auto-backgrounding) and never trips. The failing tool + // is counted instead, via background_task:failed — see + // background-failure-attribution.ts. + const relayedBackgroundFailure = + !toolSuccess && isRelayedBackgroundFailure(endEvent.toolName, errorText); + if ( + deps.toolRetryBreaker + && classifiedFailureBy !== "runtime_guard" + && !relayedBackgroundFailure + ) { const transition = deps.toolRetryBreaker.recordResult( endEvent.toolName, (sanitizedArgs ?? {}) as Record, diff --git a/packages/agent/src/executor/pi-executor/pi-executor.ts b/packages/agent/src/executor/pi-executor/pi-executor.ts index 2bfc5508e6..fdc8ab8498 100644 --- a/packages/agent/src/executor/pi-executor/pi-executor.ts +++ b/packages/agent/src/executor/pi-executor/pi-executor.ts @@ -83,6 +83,7 @@ import type { import type { CommandDirectives } from "../command-directive-types.js"; import type { StepCounter } from "../step-counter.js"; import { createToolRetryBreaker } from "../../safety/tool-retry-breaker.js"; +import { attributeBackgroundFailuresToOriginatingTool } from "../../safety/background-failure-attribution.js"; import { createMessageSendLimiter } from "../../safety/message-send-limiter.js"; import type { ComisSessionManager } from "../../session/comis-session-manager.js"; import type { RunHandle } from "../active-run-registry.js"; @@ -1370,6 +1371,7 @@ async function runSessionLocked( // the finally block at end of execute(). let cacheTrace: CacheTrace | null = null; let unsubscribeCacheTrace: (() => void) | undefined; + let unsubscribeBackgroundFailures: (() => void) | undefined; try { // Confine trajectory writes to the operator's resolved data root (so an // ancestor-symlink escape is rejected at open()) UNLESS they explicitly set @@ -1582,6 +1584,20 @@ async function runSessionLocked( suggestAlternatives: toolRetryBreakerConfig?.suggestAlternatives ?? true, }) : undefined; + // Count a BACKGROUND task's failure against the tool that launched it. + // Auto-backgrounding makes that tool report success on every launch, so + // without this its breaker never trips and a failing tool can be relaunched + // until the turn's wall-clock budget expires (see + // background-failure-attribution.ts). Same per-execution lifetime as the + // breaker it feeds — torn down in the finally below. + if (toolRetryBreaker) { + unsubscribeBackgroundFailures = attributeBackgroundFailuresToOriginatingTool({ + eventBus: deps.eventBus, + breaker: toolRetryBreaker, + ...(deps.logger === undefined ? {} : { logger: deps.logger }), + ...(deps.agentId === undefined ? {} : { agentId: deps.agentId }), + }); + } const failedToolRedirects = new Map(); // Per-execution message send limiter @@ -2583,6 +2599,7 @@ async function runSessionLocked( // failures must never throw out of finally. try { unsubscribeCacheTrace?.(); + unsubscribeBackgroundFailures?.(); } catch { // Unsubscribe failure is unreachable in practice (EventEmitter.off // is sync); swallow defensively so this never aborts cleanup. diff --git a/packages/agent/src/safety/background-failure-attribution.test.ts b/packages/agent/src/safety/background-failure-attribution.test.ts new file mode 100644 index 0000000000..c1eccdb776 --- /dev/null +++ b/packages/agent/src/safety/background-failure-attribution.test.ts @@ -0,0 +1,119 @@ +// SPDX-License-Identifier: Apache-2.0 +/** + * The breaker must count a background failure against the tool that FAILED. + * + * Live incident: a report tool was auto-backgrounded on every launch, so it + * returned "moved to the background" — a SUCCESS — and its consecutive-failure + * counter reset each time. The real failures surfaced on the `background_tasks` + * poller, which tripped at 2. The breaker therefore blinded the agent's ability + * to observe outcomes while leaving the failing tool unthrottled: 20+ launches, + * each burning the full MCP deadline, until the turn's wall-clock budget expired + * and the user got a canned error ten minutes later. + */ +import { describe, it, expect, vi } from "vitest"; +import { + attributeBackgroundFailuresToOriginatingTool, + isRelayedBackgroundFailure, + BACKGROUND_POLLER_TOOL, +} from "./background-failure-attribution.js"; + +function fakeBus() { + const handlers = new Map void>>(); + return { + on(name: string, fn: (p: unknown) => void) { + if (!handlers.has(name)) handlers.set(name, new Set()); + handlers.get(name)!.add(fn); + }, + off(name: string, fn: (p: unknown) => void) { + handlers.get(name)?.delete(fn); + }, + emit(name: string, p: unknown) { + for (const fn of handlers.get(name) ?? []) fn(p); + }, + count(name: string) { + return handlers.get(name)?.size ?? 0; + }, + }; +} + +describe("isRelayedBackgroundFailure", () => { + it("recognises the poller relaying someone else's failure", () => { + expect(isRelayedBackgroundFailure( + BACKGROUND_POLLER_TOOL, + "[conflict] Background task failed: tool timed out", + )).toBe(true); + }); + + it("still blames the poller for its OWN failures", () => { + // A bad taskId or a storage error is genuinely the poller's problem and must + // keep counting — otherwise the exemption becomes a blanket amnesty. + expect(isRelayedBackgroundFailure(BACKGROUND_POLLER_TOOL, "unknown taskId")).toBe(false); + expect(isRelayedBackgroundFailure(BACKGROUND_POLLER_TOOL, undefined)).toBe(false); + }); + + it("never exempts any other tool", () => { + expect(isRelayedBackgroundFailure("some_tool", "Background task failed: x")).toBe(false); + }); +}); + +describe("attributeBackgroundFailuresToOriginatingTool", () => { + it("counts the failure against the tool that launched the task", () => { + const bus = fakeBus(); + const breaker = { recordResult: vi.fn() }; + attributeBackgroundFailuresToOriginatingTool({ eventBus: bus as never, breaker }); + + bus.emit("background_task:failed", { + toolName: "mcp__vendor--heavy_report", + error: "timed out", + taskId: "t1", + }); + + expect(breaker.recordResult).toHaveBeenCalledTimes(1); + const [tool, , success] = breaker.recordResult.mock.calls[0]!; + expect(tool).toBe("mcp__vendor--heavy_report"); + expect(success).toBe(false); + }); + + it("repeated failures accumulate on that tool — the storm can now be stopped", () => { + const bus = fakeBus(); + const breaker = { recordResult: vi.fn() }; + attributeBackgroundFailuresToOriginatingTool({ eventBus: bus as never, breaker }); + + for (let i = 0; i < 3; i++) { + bus.emit("background_task:failed", { toolName: "mcp__vendor--heavy_report", error: "timed out" }); + } + expect(breaker.recordResult).toHaveBeenCalledTimes(3); + for (const call of breaker.recordResult.mock.calls) { + expect(call[0]).toBe("mcp__vendor--heavy_report"); + } + }); + + it("never counts the poller itself", () => { + const bus = fakeBus(); + const breaker = { recordResult: vi.fn() }; + attributeBackgroundFailuresToOriginatingTool({ eventBus: bus as never, breaker }); + bus.emit("background_task:failed", { toolName: BACKGROUND_POLLER_TOOL, error: "x" }); + expect(breaker.recordResult).not.toHaveBeenCalled(); + }); + + it("ignores another agent's failures when scoped", () => { + const bus = fakeBus(); + const breaker = { recordResult: vi.fn() }; + attributeBackgroundFailuresToOriginatingTool({ eventBus: bus as never, breaker, agentId: "a" }); + bus.emit("background_task:failed", { toolName: "t", error: "x", agentId: "b" }); + expect(breaker.recordResult).not.toHaveBeenCalled(); + bus.emit("background_task:failed", { toolName: "t", error: "x", agentId: "a" }); + expect(breaker.recordResult).toHaveBeenCalledTimes(1); + }); + + it("unsubscribes — the listener must not outlive the execution that owns the breaker", () => { + const bus = fakeBus(); + const breaker = { recordResult: vi.fn() }; + const off = attributeBackgroundFailuresToOriginatingTool({ eventBus: bus as never, breaker }); + expect(bus.count("background_task:failed")).toBe(1); + off(); + expect(bus.count("background_task:failed")).toBe(0); + bus.emit("background_task:failed", { toolName: "t", error: "x" }); + expect(breaker.recordResult).not.toHaveBeenCalled(); + }); +}); diff --git a/packages/agent/src/safety/background-failure-attribution.ts b/packages/agent/src/safety/background-failure-attribution.ts new file mode 100644 index 0000000000..b86721efe5 --- /dev/null +++ b/packages/agent/src/safety/background-failure-attribution.ts @@ -0,0 +1,118 @@ +// SPDX-License-Identifier: Apache-2.0 +/** + * Attribute a BACKGROUND task's failure to the tool that actually failed. + * + * The per-tool retry breaker is recorded against the tool that RETURNED the + * error. Auto-backgrounding breaks that assumption: it converts a slow tool into + * a non-error "moved to the background" result, so the originating tool reports + * SUCCESS on every launch and its consecutive-failure counter resets each time — + * it can never trip. The real failure surfaces later on the `background_tasks` + * POLLER, which relays it and trips instead. + * + * The consequence is the exact inverse of what a breaker is for: the agent loses + * its ability to OBSERVE results while the tool that is actually failing stays + * completely unthrottled. Live: 20+ launches of one report tool, each burning the + * full MCP call deadline, until the turn's wall-clock budget expired and the user + * got a canned error after ten minutes. + * + * This module closes the loop. `background_task:failed` already carries the + * ORIGINATING `toolName` (the manager stamps it from the task record), so the + * breaker can be told which tool to count — no parsing of the poller's prose + * error text, which names the tool only in free-form English. + * + * The other half of the fix lives at the recording site: the poller must NOT be + * blamed for relaying a failure it merely reported. See + * `isRelayedBackgroundFailure`. + * + * @module + */ + +import type { ComisLogger, TypedEventBus } from "@comis/core"; + +/** The polling tool that reports background task outcomes. Never the culprit. */ +export const BACKGROUND_POLLER_TOOL = "background_tasks"; + +/** + * The marker our own poller result carries when it relays a task failure. + * Matching on it keeps the poller's own genuine failures (a bad taskId, a + * storage error) counting against it, while a relayed task failure does not. + */ +const RELAYED_FAILURE_MARKER = "Background task failed"; + +/** + * True when this failure is the poller reporting SOMEONE ELSE's failure. + * Such a result is the poller working correctly — counting it against the + * poller trips the one tool the agent needs in order to see what went wrong. + */ +export function isRelayedBackgroundFailure( + toolName: string, + errorText: string | undefined, +): boolean { + return toolName === BACKGROUND_POLLER_TOOL + && errorText !== undefined + && errorText.includes(RELAYED_FAILURE_MARKER); +} + +/** The breaker surface this module needs (structural — no import cycle). */ +export interface BackgroundFailureBreaker { + recordResult( + toolName: string, + args: Record, + success: boolean, + errorText?: string, + ): unknown; +} + +export interface BackgroundFailureAttributionDeps { + readonly eventBus: TypedEventBus; + readonly breaker: BackgroundFailureBreaker; + readonly logger?: ComisLogger; + /** Scope to one agent; undefined attributes every agent's failures. */ + readonly agentId?: string; +} + +/** + * Subscribe so every background-task failure counts against the tool that + * launched it. Returns an unsubscribe function — callers MUST call it on + * teardown or the listener outlives the execution that owns the breaker. + * + * Args are passed empty: the breaker's same-args fingerprint is a retry signal + * for inline calls, and a backgrounded launch is already past that point. What + * matters here is the per-tool consecutive-failure count, which empty args still + * advance. + */ +export function attributeBackgroundFailuresToOriginatingTool( + deps: BackgroundFailureAttributionDeps, +): () => void { + const onFailed = (p: { + toolName?: string; + error?: string; + agentId?: string; + taskId?: string; + }): void => { + const toolName = p.toolName; + if (toolName === undefined || toolName.length === 0) return; + if (deps.agentId !== undefined && p.agentId !== undefined && p.agentId !== deps.agentId) { + return; + } + // A poller task failing is not the poller's fault either — and it is never + // the originating tool, so it would be meaningless to count. + if (toolName === BACKGROUND_POLLER_TOOL) return; + deps.breaker.recordResult(toolName, {}, false, p.error); + deps.logger?.debug( + { + step: "background-failure-attribution", + toolName, + taskId: p.taskId, + agentId: p.agentId, + hint: + "counted a background task failure against the tool that launched it — without this the tool reports success on every launch (auto-backgrounding) and its breaker never trips", + }, + "background task failure attributed to its originating tool", + ); + }; + deps.eventBus.on("background_task:failed", onFailed as never); + return () => { + deps.eventBus.off("background_task:failed", onFailed as never); + }; +} From ee4656545687745fda72b920b3284d6d52390d17 Mon Sep 17 00:00:00 2001 From: comis-agent Date: Mon, 27 Jul 2026 00:23:25 +0300 Subject: [PATCH 09/33] fix(i18n): wire the locale seam to operator config and localize the timeout reply MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two defects put the deterministic platform replies permanently in English no matter what `agents..language` was pinned to. Live: a Hebrew deployment whose agent answered in Hebrew all turn, then delivered an English canned line when the turn hit the wall-clock ceiling. 1. The pipeline-timeout reply was a hard-coded literal at the orchestrator send site, outside the degraded-reply mechanism entirely — the one message a stuck turn is guaranteed to produce was the only one no pack could reach. It is now the `pipeline_timeout` member of `LocaleMessageId`, built by `buildPipelineTimeoutReply` and carrying the incident ref like its siblings. Its text also no longer claims a transient "try again in a moment" for a turn that burned the whole ceiling and will do so again unless the ask is narrowed; it names the real cause and the actual remedy. 2. `createLocaleCatalog` was an injection seam with no production caller. The only real call site, `executor-post-execution.ts`, passed no catalog, so `DEFAULT_LOCALE_CATALOG` (no packs) resolved English for every reply — and the config reference's claim that `language` is "consumed by the deterministic degraded replies" had nothing behind it. The fix wires the seam to operator config via a new `agents..localePacks` (locale tag -> message id -> text), threaded to both send sites. The engine still ships exactly one pack, English: shipping a curated set of languages would be the runtime choosing a preferred human language, which the generic- runtime boundary forbids, and would leave every unshipped language silently English anyway. `catalogFromLocalePacks` validates ids at the boundary and logs an unknown one at WARN naming the exact config path, so a typo in a pack is never silently dead config. `PlatformReplyLocale` lives in its own leaf module: both `ExecutionPipelineDeps` and `execution-execute.ts` need the shape, and declaring it in either creates a .d.ts back-edge the cycles gate rejects. Docs corrected in the same change — both the config reference and the multilingual guide asserted `language` localized these replies. They now state that it does not, and that `localePacks` is what does. --- docs/operations/multilingual.mdx | 43 ++++++++++++--- docs/reference/config-yaml.mdx | 3 +- .../src/executor/degraded-reply-i18n.test.ts | 49 +++++++++++++++++ .../agent/src/executor/degraded-reply-i18n.ts | 54 ++++++++++++++++++- packages/agent/src/executor/degraded-reply.ts | 20 +++++++ .../executor/executor-post-execution.test.ts | 5 +- .../src/executor/executor-post-execution.ts | 21 +++++++- packages/agent/src/index.ts | 1 + .../background-failure-attribution.test.ts | 2 +- .../section-registry-parity.test.ts.snap | 47 ++++++++++++++++ .../schema-agent/schema-agent-runtime.test.ts | 34 ++++++++++++ .../schema-agent/schema-agent-runtime.ts | 9 +++- .../core/src/domain/response-locale-policy.ts | 16 ++++++ .../setup-channels/setup-channels-runtime.ts | 9 ++++ packages/orchestrator/AUDIT.md | 5 +- packages/orchestrator/src/channel-manager.ts | 3 ++ .../src/execution/execution-execute.ts | 22 +++++++- .../src/execution/execution-pipeline.test.ts | 12 ++++- .../src/execution/execution-pipeline.ts | 8 +++ .../execution-platform-reply-locale.ts | 24 +++++++++ .../src/inbound/inbound-pipeline.ts | 3 ++ .../src/inbound/setup-and-route.ts | 2 + 22 files changed, 370 insertions(+), 22 deletions(-) create mode 100644 packages/orchestrator/src/execution/execution-platform-reply-locale.ts diff --git a/docs/operations/multilingual.mdx b/docs/operations/multilingual.mdx index 6612665805..cc5c1c8ead 100644 --- a/docs/operations/multilingual.mdx +++ b/docs/operations/multilingual.mdx @@ -142,12 +142,12 @@ in mind. ## Configuration keys -Two config keys carry the multilingual surface; both are documented in the +Three config keys carry the multilingual surface; all are documented in the [config reference](/reference/config-yaml). -- **`agents..language`** — the explicit response locale, consumed by the deterministic - degraded replies (the context-exhausted and output-starved notices) and by locale-quality - observability. Accepts a **canonical BCP-47 tag** (`"he"`, `"pt-BR"`) — config validation +- **`agents..language`** — the explicit response locale, governing the model's reply + language and locale-quality observability. It does **not** translate the deterministic + platform replies; `localePacks` below does that. Accepts a **canonical BCP-47 tag** (`"he"`, `"pt-BR"`) — config validation rejects anything else. When omitted, Comis resolves the turn's locale from the channel-supplied request locale (e.g. the Telegram client's `language_code`, carried on the inbound message metadata), then from the dominant script of the current message (non-Latin @@ -172,6 +172,29 @@ Two config keys carry the multilingual surface; both are documented in the message looks like, set `agents..language`; without it, per-message language switching is followed naturally but never forced. See the [`agents` reference](/reference/config-yaml#agents). +- **`agents..localePacks`** — the strings for the **deterministic platform replies**: the + canned lines the runtime itself sends when a turn degrades (wall-clock timeout, context + exhausted, output truncated, loop halted). These are not model output, so `language` cannot + reach them — they come from a string table, and the engine ships exactly one pack, English. + Supply your locale's strings here: + + ```yaml + agents: + default: + language: he + localePacks: + he: + pipeline_timeout: "עצרתי את הבקשה — היא ארכה יותר מהזמן המוקצב לתור אחד." + loop_detected: "עצרתי כי חזרתי על אותה פעולה ללא התקדמות." + ``` + + A locale variant falls back to its base language (`he-IL` → `he`), and any message id you omit + falls back to English — so a partial pack is safe. An id the runtime does not recognize is + ignored and logged at WARN naming the exact config path, so a typo never becomes silently dead + config. Ids and full details: [`config` reference](/reference/config-yaml#locale-packs). + + **Setting `language: he` alone leaves these replies in English.** That is the single most + common surprise here: the model answers in Hebrew while a timeout notice arrives in English. - **`embedding.multilingual`** — an advisory boolean for the `comis system-health` model-health line (see [Embedding and reranker](#embedding-and-reranker)). It does not gate search. See the [configuration reference](/reference/config-yaml). @@ -264,10 +287,14 @@ resistance on the recommended local model). Translating regex patterns into N la maintenance treadmill with near-zero marginal coverage. This stance is documented here so it is not mistaken for an oversight. -**No-translation principle.** Comis never translates user content. Summaries, memories, and -deterministic replies stay in the conversation's language, and code identifiers, file paths, tool -names, configuration knob paths, and trace ids stay **verbatim in every language** — a Hebrew -degraded reply still names `contextEngine.budget.effectiveContextCapSmall` and the trace id exactly. +**No-translation principle.** Comis never translates user content. Summaries and memories stay +in the conversation's language, and code identifiers, file paths, tool names, configuration knob +paths, and trace ids stay **verbatim in every language** — a Hebrew degraded reply still names +`contextEngine.budget.effectiveContextCapSmall` and the trace id exactly. The same principle is +why the runtime does not translate its **own** deterministic replies either: it ships one pack, +English, and a deployment supplies the rest through `localePacks`. Shipping a curated set of +languages in the engine would be the runtime choosing a preferred human language, which it must +not do — and it would leave every unshipped language silently English. ## Local and small models (non-Latin) diff --git a/docs/reference/config-yaml.mdx b/docs/reference/config-yaml.mdx index 5fe8b781e7..28ad3a3d11 100644 --- a/docs/reference/config-yaml.mdx +++ b/docs/reference/config-yaml.mdx @@ -257,7 +257,8 @@ Multi-agent configuration map. Each key is an agent ID, and the value is a `PerA | `preserveRecent` | `number` | `4` | Minimum recent messages to always preserve during compaction | | `workspacePath` | `string` | _(unset)_ | Path to agent workspace directory containing identity files | | `reactionLevel` | `enum` | _(unset)_ | Reaction frequency: `minimal` (1 per 5-10 exchanges) or `extensive` (react freely) | -| `language` | `string` | _(unset)_ | Explicit response locale, consumed by the deterministic degraded replies (context-exhausted / output-starved notices) and locale-quality observability. Must be a canonical BCP-47 tag (`"he"`, `"pt-BR"`) — validation rejects anything else. When omitted, Comis resolves the turn's locale from the channel-supplied request locale (e.g. the Telegram client's `language_code` — ignored for the turn when the message's script contradicts it), then the dominant non-Latin script of the current message (as a script-only locale such as `und-Hebr`), else leaves it unset. **Only this explicit key ENFORCES the reply language**: when it is set, a script-mismatched final response triggers at most one bounded tools-disabled repair turn. Both inferred tiers are advisory — they inform the prompt but never trigger a repair, so a client's UI language cannot override the language the conversation is actually being held in. Set this key when you need a fixed reply language regardless of any individual message (see [Multilingual](/operations/multilingual)). | +| `language` | `string` | _(unset)_ | Explicit response locale, governing the LLM's reply language and locale-quality observability. It does **not** by itself translate the deterministic platform replies (the canned timeout / context-exhausted / output-starved / loop-halted notices) — those come from a string table, and the runtime ships only English. Supply the strings for your language in [`localePacks`](#locale-packs) alongside this key. Must be a canonical BCP-47 tag (`"he"`, `"pt-BR"`) — validation rejects anything else. When omitted, Comis resolves the turn's locale from the channel-supplied request locale (e.g. the Telegram client's `language_code` — ignored for the turn when the message's script contradicts it), then the dominant non-Latin script of the current message (as a script-only locale such as `und-Hebr`), else leaves it unset. **Only this explicit key ENFORCES the reply language**: when it is set, a script-mismatched final response triggers at most one bounded tools-disabled repair turn. Both inferred tiers are advisory — they inform the prompt but never trigger a repair, so a client's UI language cannot override the language the conversation is actually being held in. Set this key when you need a fixed reply language regardless of any individual message (see [Multilingual](/operations/multilingual)). | +| `localePacks` | `object` | _(unset)_ | Operator strings for the deterministic platform replies, keyed locale tag → message id → text: `localePacks: { he: { pipeline_timeout: "..." } }`. The runtime holds no human language but English, so this is the only way those replies match the language your users are answered in. Locale keys are canonical BCP-47 (same validation as `language`); a variant falls back to its base language (`he-IL` → `he`), and any id you do not supply falls back to English. Valid ids: `pipeline_timeout`, `context_exhausted`, `output_starved`, `loop_detected`, `cause_oversized_input`, `cause_oversized_history_message`, `cause_fixed_overhead_exceeds_window`, `advice_default`, `advice_history`, `advice_fixed_overhead`. An unrecognized id is ignored and logged at WARN naming the exact path, so a typo is never silently dead config. | | `enforceFinalTag` | `boolean` | `false` | When enabled, only content inside `` blocks reaches users. Suppresses all content outside final tags on both streaming and non-streaming paths. | | `fastMode` | `boolean` | `false` | Enables fast mode for the LLM provider (provider-specific behavior). | | `storeCompletions` | `boolean` | `false` | When enabled, sends `store: true` to OpenAI-compatible providers for completion storage. | diff --git a/packages/agent/src/executor/degraded-reply-i18n.test.ts b/packages/agent/src/executor/degraded-reply-i18n.test.ts index 89ffa575b2..cf1e02bff0 100644 --- a/packages/agent/src/executor/degraded-reply-i18n.test.ts +++ b/packages/agent/src/executor/degraded-reply-i18n.test.ts @@ -1,10 +1,12 @@ // SPDX-License-Identifier: Apache-2.0 import { describe, expect, it } from "vitest"; import { + catalogFromLocalePacks, createLocaleCatalog, selectContextExhaustedReply, selectLoopDetectedReply, selectOutputStarvedAnnotation, + selectPipelineTimeoutReply, } from "./degraded-reply-i18n.js"; describe("extensible locale catalog", () => { @@ -39,3 +41,50 @@ describe("extensible locale catalog", () => { .not.toContain("contextEngine."); }); }); + +// The pipeline timeout is the LAST thing a user sees on a turn that ran to the +// wall-clock ceiling. It was a hard-coded English literal at the send site, +// outside this mechanism entirely — so a deployment answering in another +// language still got English, and no pack could ever reach it. +describe("pipeline timeout reply", () => { + it("is a member of the localizable platform-reply set", () => { + const catalog = createLocaleCatalog({ "fr-CA": { pipeline_timeout: "localized timeout" } }); + expect(selectPipelineTimeoutReply("fr-CA", {}, catalog)).toBe("localized timeout"); + }); + + it("falls back to the English pack with no locale or no matching pack", () => { + expect(selectPipelineTimeoutReply(undefined, {})).toContain("taking too long"); + expect(selectPipelineTimeoutReply("fr-CA", {}, createLocaleCatalog({}))).toContain( + "taking too long", + ); + }); + + it("appends the incident ref so `comis explain` is one step from the chat", () => { + expect(selectPipelineTimeoutReply(undefined, { traceId: "t-1" })).toContain("(incident t-1)"); + }); +}); + +// Operator config carries packs as an open string->string record: core cannot +// own the message-id list without the runtime's reply vocabulary leaking into +// it. The runtime therefore validates the ids at the boundary. +describe("catalogFromLocalePacks", () => { + it("builds a working catalog from raw operator config", () => { + const catalog = catalogFromLocalePacks({ he: { pipeline_timeout: "בקשה ארכה מדי" } }); + expect(selectPipelineTimeoutReply("he", {}, catalog)).toBe("בקשה ארכה מדי"); + }); + + it("returns the English-only default catalog for undefined or empty packs", () => { + expect(selectPipelineTimeoutReply("he", {}, catalogFromLocalePacks(undefined))) + .toContain("taking too long"); + }); + + it("drops unknown message ids instead of silently carrying dead config", () => { + const onUnknown: string[] = []; + const catalog = catalogFromLocalePacks( + { he: { pipeline_timeout: "ok", not_a_message_id: "dead" } }, + (locale, id) => onUnknown.push(`${locale}:${id}`), + ); + expect(selectPipelineTimeoutReply("he", {}, catalog)).toBe("ok"); + expect(onUnknown).toEqual(["he:not_a_message_id"]); + }); +}); diff --git a/packages/agent/src/executor/degraded-reply-i18n.ts b/packages/agent/src/executor/degraded-reply-i18n.ts index cbce91e7b7..4bd998c30e 100644 --- a/packages/agent/src/executor/degraded-reply-i18n.ts +++ b/packages/agent/src/executor/degraded-reply-i18n.ts @@ -16,7 +16,8 @@ export type LocaleMessageId = | "advice_history" | "advice_fixed_overhead" | "output_starved" - | "loop_detected"; + | "loop_detected" + | "pipeline_timeout"; export type LocalePack = Readonly>>; @@ -44,6 +45,11 @@ const ENGLISH_PACK: Readonly> = { "I stopped because I kept repeating an action that wasn't making progress " + "(usually a tool that failed or was blocked) and didn't want to loop. The " + "request may need a different approach, or that capability isn't available here.", + pipeline_timeout: + "I stopped this request because it was taking too long and hit the time limit " + + "for a single turn. Nothing was left half-applied. If it needs many lookups, " + + "ask for a narrower slice (fewer items, a shorter date range) and I can do the " + + "rest in follow-ups.", }; function canonicalLocale(raw: string): string | undefined { @@ -80,6 +86,39 @@ export function createLocaleCatalog( export const DEFAULT_LOCALE_CATALOG = createLocaleCatalog(); +/** Every id an operator pack may define. Exported so config surfaces can list them. */ +export const LOCALE_MESSAGE_IDS: readonly LocaleMessageId[] = Object.keys( + ENGLISH_PACK, +) as LocaleMessageId[]; + +const KNOWN_MESSAGE_IDS = new Set(LOCALE_MESSAGE_IDS); + +/** + * Build a catalog from the operator's raw `localePacks` config. + * + * This is the seam's ONLY production entry point. `createLocaleCatalog` takes a + * typed pack; operator config arrives as an open `string -> string` record + * because core does not own this runtime's message-id vocabulary. Unknown ids + * are dropped and reported rather than silently retained — a typo in a pack + * would otherwise look configured while the reply stayed English. + */ +export function catalogFromLocalePacks( + packs: Readonly>>> | undefined, + onUnknownId?: (locale: string, messageId: string) => void, +): LocaleCatalog { + if (packs === undefined) return DEFAULT_LOCALE_CATALOG; + const typed: Record = {}; + for (const [locale, pack] of Object.entries(packs)) { + const known: Partial> = {}; + for (const [id, text] of Object.entries(pack)) { + if (KNOWN_MESSAGE_IDS.has(id)) known[id as LocaleMessageId] = text; + else onUnknownId?.(locale, id); + } + if (Object.keys(known).length > 0) typed[locale] = known; + } + return createLocaleCatalog(typed); +} + function incidentRef(traceId?: string): string { return traceId !== undefined && traceId.length > 0 ? ` (incident ${traceId})` : ""; } @@ -136,3 +175,16 @@ export function selectLoopDetectedReply( ): string { return catalog.resolve(locale, "loop_detected") + incidentRef(opts.traceId); } + +/** + * The reply for a turn killed by the execution wall-clock ceiling + * (`executionTimeoutMs`). The model never returned, so there is no partial text + * to annotate — this REPLACES the response entirely. + */ +export function selectPipelineTimeoutReply( + locale: string | undefined, + opts: { traceId?: string }, + catalog: LocaleCatalog = DEFAULT_LOCALE_CATALOG, +): string { + return catalog.resolve(locale, "pipeline_timeout") + incidentRef(opts.traceId); +} diff --git a/packages/agent/src/executor/degraded-reply.ts b/packages/agent/src/executor/degraded-reply.ts index ea78651da4..93354d4dbd 100644 --- a/packages/agent/src/executor/degraded-reply.ts +++ b/packages/agent/src/executor/degraded-reply.ts @@ -23,9 +23,13 @@ import { selectOutputStarvedAnnotation, selectContextExhaustedReply, selectLoopDetectedReply, + selectPipelineTimeoutReply, type LocaleCatalog, } from "./degraded-reply-i18n.js"; +export { catalogFromLocalePacks, LOCALE_MESSAGE_IDS } from "./degraded-reply-i18n.js"; +export type { LocaleCatalog, LocaleMessageId, LocalePack } from "./degraded-reply-i18n.js"; + // CAP_KNOB_BY_CLASS lives in degraded-reply-i18n.ts as an internal diagnostic // mapping. Re-exported here for callers that need to associate capability // classes with operator settings; user-facing replies do not interpolate it. @@ -104,3 +108,19 @@ export function buildLoopDetectedReply(opts?: ContextExhaustedReplyOpts): string traceId: opts?.traceId, }, opts?.localeCatalog); } + +/** + * Honest reply for a turn the execution wall-clock ceiling killed + * (`executionTimeoutMs`). REPLACES the response — a pipeline timeout means the + * model never returned, so there is nothing partial to annotate. + * + * This exists so the timeout reply is a MEMBER of the localizable platform-reply + * set. It used to be a literal at the send site in the orchestrator, which put + * the one message a stuck turn is guaranteed to produce outside the only + * mechanism that can translate it. PURE: same opts → same string. + */ +export function buildPipelineTimeoutReply(opts?: ContextExhaustedReplyOpts): string { + return selectPipelineTimeoutReply(opts?.language, { + traceId: opts?.traceId, + }, opts?.localeCatalog); +} diff --git a/packages/agent/src/executor/executor-post-execution.test.ts b/packages/agent/src/executor/executor-post-execution.test.ts index 04b6a7362e..a899728679 100644 --- a/packages/agent/src/executor/executor-post-execution.test.ts +++ b/packages/agent/src/executor/executor-post-execution.test.ts @@ -2505,8 +2505,9 @@ describe("degraded-reply chokepoint consumes the typed locale policy", () => { it("source-grep — the resolved tag reaches all three builders (language passed in)", () => { const block = readDegradedBlock(); - // output_starved: buildOutputStarvedAnnotation() — called with an argument. - expect(block).toMatch(/buildOutputStarvedAnnotation\(\s*[A-Za-z_$][\w$]*\s*\)/); + // output_starved: buildOutputStarvedAnnotation(, ) — the tag + // is the first argument; the operator-config catalog rides alongside it. + expect(block).toMatch(/buildOutputStarvedAnnotation\(\s*[A-Za-z_$][\w$]*\s*,/); // context_exhausted + loop_detected: a `language:` field in the opts object. const languageFields = block.match(/\blanguage\s*:/g) ?? []; // At least the context_exhausted and loop_detected opts carry `language:`. diff --git a/packages/agent/src/executor/executor-post-execution.ts b/packages/agent/src/executor/executor-post-execution.ts index 2a681aae01..c46d24994d 100644 --- a/packages/agent/src/executor/executor-post-execution.ts +++ b/packages/agent/src/executor/executor-post-execution.ts @@ -121,7 +121,7 @@ import { createHash, randomUUID } from "node:crypto"; // Critic hook (no inline logic — all logic in verification-gate.ts) import { shouldRunCritic, runVerificationCritic } from "./verification-gate.js"; // Deterministic user-facing replies for named degraded terminal causes. -import { buildOutputStarvedAnnotation, buildContextExhaustedReply, buildLoopDetectedReply } from "./degraded-reply.js"; +import { buildOutputStarvedAnnotation, buildContextExhaustedReply, buildLoopDetectedReply, catalogFromLocalePacks, LOCALE_MESSAGE_IDS } from "./degraded-reply.js"; import { parseContextExhaustionCause } from "../context-engine/errors.js"; import { buildSyntheticCriticDeps } from "./verification-gate-synth-deps.js"; import { resolveScaffoldDefaults } from "./scaffold-defaults.js"; @@ -1395,8 +1395,23 @@ export async function postExecution(params: PostExecutionParams): Promise // each deterministic degraded-reply builder. Missing locale packs fall back // to the injected catalog's English strings. const replyLanguage = params.responseLocalePolicy.locale; + // Wire the locale seam to operator config. `createLocaleCatalog` had exactly + // one production caller — the no-packs DEFAULT_LOCALE_CATALOG — so every + // deterministic reply resolved English no matter what `language` was pinned, + // and the documented "consumed by the deterministic degraded replies" claim + // had nothing behind it. An unknown id is reported, never silently kept. + const localeCatalog = catalogFromLocalePacks(config.localePacks, (locale, messageId) => { + deps.logger.warn( + { + step: "degraded-reply", + errorKind: "config" as const, + hint: `agents..localePacks.${locale}.${messageId} is not a platform-reply message id, so it is ignored; valid ids: ${LOCALE_MESSAGE_IDS.join(", ")}`, + }, + "unknown locale pack message id ignored", + ); + }); if (effectiveFinishReason === "output_starved") { - result.response = (result.response ?? "") + buildOutputStarvedAnnotation(replyLanguage); + result.response = (result.response ?? "") + buildOutputStarvedAnnotation(replyLanguage, localeCatalog); deps.logger.warn( { step: "degraded-reply", errorKind: "resource" as const, hint: "output_starved annotation appended" }, "output_starved — annotated truncated reply", @@ -1420,6 +1435,7 @@ export async function postExecution(params: PostExecutionParams): Promise ...(incidentTraceId !== undefined ? { traceId: incidentTraceId } : {}), cause: exhaustionCause, language: replyLanguage, + localeCatalog, }); deps.logger.warn( { step: "degraded-reply", errorKind: "resource" as const, hint: "context_exhausted synthesized reply" }, @@ -1435,6 +1451,7 @@ export async function postExecution(params: PostExecutionParams): Promise const loopReply = buildLoopDetectedReply({ ...(loopTraceId !== undefined ? { traceId: loopTraceId } : {}), language: replyLanguage, + localeCatalog, }); result.response = existing.length > 0 ? `${existing}\n\n${loopReply}` : loopReply; deps.logger.warn( diff --git a/packages/agent/src/index.ts b/packages/agent/src/index.ts index b3355f0135..5892b1eded 100644 --- a/packages/agent/src/index.ts +++ b/packages/agent/src/index.ts @@ -539,6 +539,7 @@ export type { CodeRegion } from "./response-filter/code-regions.js"; // Thinking tag filter export { createThinkingTagFilter } from "./response-filter/thinking-tag-filter.js"; +export { buildPipelineTimeoutReply, catalogFromLocalePacks } from "./executor/degraded-reply.js"; export type { ThinkingTagFilter, ThinkingTagFilterOptions } from "./response-filter/thinking-tag-filter.js"; // Operation model resolver diff --git a/packages/agent/src/safety/background-failure-attribution.test.ts b/packages/agent/src/safety/background-failure-attribution.test.ts index c1eccdb776..deeaebc8f6 100644 --- a/packages/agent/src/safety/background-failure-attribution.test.ts +++ b/packages/agent/src/safety/background-failure-attribution.test.ts @@ -37,7 +37,7 @@ function fakeBus() { } describe("isRelayedBackgroundFailure", () => { - it("recognises the poller relaying someone else's failure", () => { + it("returns true when the poller relays someone else's failure", () => { expect(isRelayedBackgroundFailure( BACKGROUND_POLLER_TOOL, "[conflict] Background task failed: tool timed out", diff --git a/packages/core/src/config/__snapshots__/section-registry-parity.test.ts.snap b/packages/core/src/config/__snapshots__/section-registry-parity.test.ts.snap index b67d98ddae..1ee22c98d5 100644 --- a/packages/core/src/config/__snapshots__/section-registry-parity.test.ts.snap +++ b/packages/core/src/config/__snapshots__/section-registry-parity.test.ts.snap @@ -1465,6 +1465,11 @@ exports[`section-registry parity > field-metadata view > getFieldMetadata("agent "path": "agents.learningOutcome.sources", "type": "array" }, + { + "immutable": true, + "path": "agents.localePacks", + "type": "object" + }, { "default": 100000, "immutable": true, @@ -17900,6 +17905,27 @@ exports[`section-registry parity > schema-serializer view > getConfigSchema("age ], "type": "object" }, + "localePacks": { + "additionalProperties": { + "additionalProperties": { + "maxLength": 2000, + "minLength": 1, + "type": "string" + }, + "propertyNames": { + "maxLength": 64, + "minLength": 1, + "type": "string" + }, + "type": "object" + }, + "propertyNames": { + "maxLength": 128, + "minLength": 2, + "type": "string" + }, + "type": "object" + }, "maxContextChars": { "default": 100000, "exclusiveMinimum": 0, @@ -29133,6 +29159,27 @@ exports[`section-registry parity > schema-serializer view > getConfigSchema() ], "type": "object" }, + "localePacks": { + "additionalProperties": { + "additionalProperties": { + "maxLength": 2000, + "minLength": 1, + "type": "string" + }, + "propertyNames": { + "maxLength": 64, + "minLength": 1, + "type": "string" + }, + "type": "object" + }, + "propertyNames": { + "maxLength": 128, + "minLength": 2, + "type": "string" + }, + "type": "object" + }, "maxContextChars": { "default": 100000, "exclusiveMinimum": 0, diff --git a/packages/core/src/config/schema-agent/schema-agent-runtime.test.ts b/packages/core/src/config/schema-agent/schema-agent-runtime.test.ts index bbf9ab60d5..395f62fb09 100644 --- a/packages/core/src/config/schema-agent/schema-agent-runtime.test.ts +++ b/packages/core/src/config/schema-agent/schema-agent-runtime.test.ts @@ -163,6 +163,40 @@ describe("agents..language config key", () => { }); }); +// --------------------------------------------------------------------------- +// agents..localePacks — operator strings for the deterministic platform +// replies. The runtime ships English only; `language` alone cannot translate +// them, so this is the key that actually makes a non-English deployment's +// degraded replies match the language its users are answered in. +// --------------------------------------------------------------------------- +describe("agents..localePacks config key", () => { + it("is optional on a bare config", () => { + expect(PerAgentConfigSchema.parse({}).localePacks).toBeUndefined(); + }); + + it("accepts locale-keyed message-id maps for any canonical tag", () => { + const cfg = PerAgentConfigSchema.parse({ + localePacks: { he: { pipeline_timeout: "timed out", loop_detected: "looped" } }, + }); + expect(cfg.localePacks?.he?.pipeline_timeout).toBe("timed out"); + }); + + it("rejects non-canonical locale keys the same way `language` does", () => { + expect( + PerAgentConfigSchema.safeParse({ localePacks: { French: { loop_detected: "x" } } }).success, + ).toBe(false); + }); + + it("rejects empty and non-string message values", () => { + expect( + PerAgentConfigSchema.safeParse({ localePacks: { he: { loop_detected: "" } } }).success, + ).toBe(false); + expect( + PerAgentConfigSchema.safeParse({ localePacks: { he: { loop_detected: 42 } } }).success, + ).toBe(false); + }); +}); + // --------------------------------------------------------------------------- // agents..capabilityClass — // an operator pin for the capability class. Without it only codex models are diff --git a/packages/core/src/config/schema-agent/schema-agent-runtime.ts b/packages/core/src/config/schema-agent/schema-agent-runtime.ts index 90ada017ed..01b711878c 100644 --- a/packages/core/src/config/schema-agent/schema-agent-runtime.ts +++ b/packages/core/src/config/schema-agent/schema-agent-runtime.ts @@ -28,7 +28,7 @@ import { LearningOutcomeConfigSchema } from "../schema-learning-outcome.js"; import { LearningConfigSchema } from "../schema-learning.js"; import { MemoryLifecycleConfigSchema } from "../schema-memory-lifecycle.js"; import { validateProfileId } from "../../security/profile-id.js"; -import { CanonicalLocaleSchema } from "../../domain/response-locale-policy.js"; +import { CanonicalLocaleSchema, LocalePacksSchema } from "../../domain/response-locale-policy.js"; import { ChannelEndpointSchema } from "../../domain/conversation-scope.js"; // Sibling-leaf imports (one-directional dependency graph). @@ -195,6 +195,13 @@ export const AgentConfigSchema = z.strictObject({ operationModels: OperationModelsSchema, /** Canonical BCP-47 locale for response policy and deterministic platform replies. */ language: CanonicalLocaleSchema.optional(), + /** + * Strings for the deterministic platform replies, per locale. Without an + * entry for the resolved `language`, those replies fall back to English — + * the only pack the runtime ships. Setting `language` alone does NOT + * translate them; it cannot, because the runtime holds no other language. + */ + localePacks: LocalePacksSchema.optional(), }); export type AgentConfig = z.infer; diff --git a/packages/core/src/domain/response-locale-policy.ts b/packages/core/src/domain/response-locale-policy.ts index 3e51c9e3d2..7adc6a75c4 100644 --- a/packages/core/src/domain/response-locale-policy.ts +++ b/packages/core/src/domain/response-locale-policy.ts @@ -12,6 +12,22 @@ export const CanonicalLocaleSchema = z.string().trim().min(2).max(128).refine( "locale must be a canonical BCP-47 language tag", ); +/** + * Operator-supplied strings for the deterministic platform replies (the canned + * lines the runtime sends when a turn degrades — timeout, context exhausted, + * output truncated, loop halted). Keyed locale tag → message id → string. + * + * The runtime ships ONE pack, English, as its fallback. It does not know any + * other human language and must not: a deployment that answers its users in + * another language supplies that language's strings here. Message ids are + * validated by the consuming runtime, not here — core deliberately does not + * own that list. + */ +export const LocalePacksSchema = z.record( + CanonicalLocaleSchema, + z.record(z.string().min(1).max(64), z.string().min(1).max(2000)), +); + export const ResponseLocaleSourceSchema = z.enum([ "request", "explicit", diff --git a/packages/daemon/src/wiring/setup-channels/setup-channels-runtime.ts b/packages/daemon/src/wiring/setup-channels/setup-channels-runtime.ts index 66cb923d8a..9edeee298d 100644 --- a/packages/daemon/src/wiring/setup-channels/setup-channels-runtime.ts +++ b/packages/daemon/src/wiring/setup-channels/setup-channels-runtime.ts @@ -305,6 +305,15 @@ export async function buildAndStartChannelManager( const agentConfig = agents[agentId]; return agentConfig?.elevatedReply; }, + getPlatformReplyLocale: (agentId: string) => { + const agentConfig = agents[agentId]; + return { + ...(agentConfig?.language === undefined ? {} : { language: agentConfig.language }), + ...(agentConfig?.localePacks === undefined + ? {} + : { localePacks: agentConfig.localePacks }), + }; + }, getEnforceFinalTag: (agentId: string) => { const agentConfig = agents[agentId]; return agentConfig?.enforceFinalTag; diff --git a/packages/orchestrator/AUDIT.md b/packages/orchestrator/AUDIT.md index 4a778a033c..ea44a01fd4 100644 --- a/packages/orchestrator/AUDIT.md +++ b/packages/orchestrator/AUDIT.md @@ -1,9 +1,9 @@ # ChannelManagerDeps Audit **Generated:** 2026-05-11 -**Interface source:** `packages/orchestrator/src/channel-manager.ts` (50-field interface) +**Interface source:** `packages/orchestrator/src/channel-manager.ts` (51-field interface) **Construction site:** `packages/daemon/src/wiring/setup-channels/setup-channels-runtime.ts` (single site — `createChannelManager({`) -**Field count:** 50 (13 required + 37 optional + 0 stale-fallback) +**Field count:** 51 (13 required + 38 optional + 0 stale-fallback) This file is co-located with the orchestrator package. `files: ["dist"]` in `packages/orchestrator/package.json` excludes it from the npm tarball. @@ -43,6 +43,7 @@ The table below uses a tight Markdown shape — `| | ElevatedReplyConfig | undefined; + /** Agent locale pin + operator strings for the deterministic platform replies. */ + getPlatformReplyLocale?: (agentId: string) => PlatformReplyLocale | undefined; /** Optional tool assembler for resolving agent tools before execution. When absent, executor receives no tools (undefined). * The optional `options` object carries per-call wiring -- currently used to thread the inbound session's * structural SessionKey so the assembled tools resolve the session-lifetime FileStateTracker via diff --git a/packages/orchestrator/src/execution/execution-execute.ts b/packages/orchestrator/src/execution/execution-execute.ts index 889c9ed42f..0cad3e2e4c 100644 --- a/packages/orchestrator/src/execution/execution-execute.ts +++ b/packages/orchestrator/src/execution/execution-execute.ts @@ -18,9 +18,15 @@ import type { InboundMessageProvenancePlan, } from "@comis/agent"; import type { CommandDirectives } from "../commands/index.js"; -import { sanitizeAssistantResponse, createThinkingTagFilter } from "@comis/agent"; +import { + sanitizeAssistantResponse, + createThinkingTagFilter, + buildPipelineTimeoutReply, + catalogFromLocalePacks, +} from "@comis/agent"; import type { ExecutionPipelineDeps } from "./execution-pipeline.js"; +export type { PlatformReplyLocale } from "./execution-platform-reply-locale.js"; import { emitObservationalEvent } from "./execution-event-emitter.js"; import { buildThreadSendOpts } from "./execution-routing-config.js"; import type { TypingLifecycleController } from "@comis/channels"; @@ -33,6 +39,7 @@ import type { TypingLifecycleController } from "@comis/channels"; export type ExecuteDeps = Pick< ExecutionPipelineDeps, "eventBus" | "logger" | "assembleToolsForAgent" | "executionTimeoutMs" | "enforceFinalTag" + | "getPlatformReplyLocale" >; // --------------------------------------------------------------------------- @@ -227,9 +234,20 @@ export async function executeLlm( timestamp: systemNowMs(), }); + // Route the reply through the deterministic platform-reply mechanism + // instead of a literal. This is the ONE message a wall-clock-killed turn + // is guaranteed to produce, and it was the only degraded reply outside + // that mechanism — so it could not be localized at all, and it claimed a + // transient "try again in a moment" for a turn that had burned the full + // ceiling and would do so again unless the ask were narrowed. + const replyLocale = deps.getPlatformReplyLocale?.(agentId); await adapter.sendMessage( effectiveMsg.channelId, - "I'm having trouble processing your request right now. Please try again in a moment.", + buildPipelineTimeoutReply({ + ...(replyLocale?.language === undefined ? {} : { language: replyLocale.language }), + localeCatalog: catalogFromLocalePacks(replyLocale?.localePacks), + traceId: executionTraceId, + }), { replyTo, ...buildThreadSendOpts(effectiveMsg.metadata) }, // eslint-disable-next-line no-restricted-syntax -- intentional fire-and-forget ).catch(() => { /* adapter logs internally */ }); diff --git a/packages/orchestrator/src/execution/execution-pipeline.test.ts b/packages/orchestrator/src/execution/execution-pipeline.test.ts index f64f36912d..78482266e9 100644 --- a/packages/orchestrator/src/execution/execution-pipeline.test.ts +++ b/packages/orchestrator/src/execution/execution-pipeline.test.ts @@ -1518,12 +1518,20 @@ describe("executeAndDeliver", () => { makeBlockStreamCfg(), new Set(), makeSendOverrides(), ); - // Canned error sent to channel + // The timeout reply goes out through the deterministic platform-reply + // mechanism (so an operator pack can localize it), names the real cause, + // and no longer promises a transient "try again in a moment" for a turn + // that burned the whole ceiling. expect(adapter.sendMessage).toHaveBeenCalledWith( "12345", - "I'm having trouble processing your request right now. Please try again in a moment.", + expect.stringContaining("hit the time limit"), expect.objectContaining({}), ); + expect(adapter.sendMessage).not.toHaveBeenCalledWith( + "12345", + expect.stringContaining("try again in a moment"), + expect.anything(), + ); // execution:aborted emitted with pipeline_timeout reason expect(eventBus.emit).toHaveBeenCalledWith( diff --git a/packages/orchestrator/src/execution/execution-pipeline.ts b/packages/orchestrator/src/execution/execution-pipeline.ts index 170937ca66..9052b11642 100644 --- a/packages/orchestrator/src/execution/execution-pipeline.ts +++ b/packages/orchestrator/src/execution/execution-pipeline.ts @@ -39,6 +39,7 @@ import type { import type { RetryEngine } from "@comis/core"; import { executeLlm } from "./execution-execute.js"; +import type { PlatformReplyLocale } from "./execution-platform-reply-locale.js"; import { filterExecutionResponse, } from "./execution-filter.js"; @@ -83,6 +84,13 @@ export interface ExecutionPipelineDeps { streamingConfig?: StreamingConfig; sendPolicyConfig?: SendPolicyConfig; getElevatedReplyConfig?: (agentId: string) => ElevatedReplyConfig | undefined; + /** + * The agent's locale pin + operator-supplied strings for the deterministic + * platform replies. Needed here because the pipeline-timeout reply is sent + * from this layer, after the executor (which owns every other degraded reply) + * has already been abandoned by `withTimeout`. + */ + getPlatformReplyLocale?: (agentId: string) => PlatformReplyLocale | undefined; channelRegistry?: ChannelRegistry; retryEngine?: RetryEngine; commandQueue?: CommandQueue; diff --git a/packages/orchestrator/src/execution/execution-platform-reply-locale.ts b/packages/orchestrator/src/execution/execution-platform-reply-locale.ts new file mode 100644 index 0000000000..48126759c6 --- /dev/null +++ b/packages/orchestrator/src/execution/execution-platform-reply-locale.ts @@ -0,0 +1,24 @@ +// SPDX-License-Identifier: Apache-2.0 +/** + * The accessor shape for an agent's deterministic-platform-reply locale. + * + * A LEAF with no imports, deliberately. `ExecutionPipelineDeps` (the deps owner) + * and `execution-execute.ts` (the consumer, which already imports the deps type + * from the pipeline) both need this shape, so declaring it in either one puts a + * back-edge between them — a real `.d.ts` cycle the `cycles` gate rejects. + * + * Structural on purpose: this layer sends the pipeline-timeout reply but must + * not depend on the config package to describe where its strings come from. + * + * @module + */ + +/** + * An agent's response-locale pin plus the operator-supplied strings for the + * deterministic platform replies (locale tag → message id → text). Both + * optional: with neither, those replies use the runtime's English pack. + */ +export interface PlatformReplyLocale { + readonly language?: string; + readonly localePacks?: Readonly>>>; +} diff --git a/packages/orchestrator/src/inbound/inbound-pipeline.ts b/packages/orchestrator/src/inbound/inbound-pipeline.ts index 4ad3970df0..ce02f5f5bc 100644 --- a/packages/orchestrator/src/inbound/inbound-pipeline.ts +++ b/packages/orchestrator/src/inbound/inbound-pipeline.ts @@ -9,6 +9,7 @@ */ import type { AgentExecutor, InboundMessageProvenancePlan } from "@comis/agent"; +import type { PlatformReplyLocale } from "../execution/execution-platform-reply-locale.js"; // Relative path used because orchestrator cannot import its own published name. import type { MessageRouter } from "../routing/message-router.js"; import type { SessionLifecycle } from "@comis/agent"; @@ -101,6 +102,8 @@ export interface InboundPipelineDeps { getResetTriggers?: (agentId: string) => string[]; queueConfig?: QueueConfig; getElevatedReplyConfig?: (agentId: string) => ElevatedReplyConfig | undefined; + /** See ChannelManagerDeps. */ + getPlatformReplyLocale?: (agentId: string) => PlatformReplyLocale | undefined; // eslint-disable-next-line @typescript-eslint/no-explicit-any assembleToolsForAgent?: (agentId: string, options?: { sessionKey?: SessionKey }) => Promise; audioPreflight?: (msg: NormalizedMessage) => Promise; diff --git a/packages/orchestrator/src/inbound/setup-and-route.ts b/packages/orchestrator/src/inbound/setup-and-route.ts index 98f7bc94f7..0f5f559b1b 100644 --- a/packages/orchestrator/src/inbound/setup-and-route.ts +++ b/packages/orchestrator/src/inbound/setup-and-route.ts @@ -101,6 +101,7 @@ export type SetupAndRouteDeps = Pick< | "sessionResolver" | "sendPolicyConfig" | "getElevatedReplyConfig" + | "getPlatformReplyLocale" | "retryEngine" | "deliveryQueue" | "deliveryService" @@ -306,6 +307,7 @@ export async function setupAndRoute( streamingConfig: deps.streamingConfig, sendPolicyConfig: deps.sendPolicyConfig, getElevatedReplyConfig: deps.getElevatedReplyConfig, + getPlatformReplyLocale: deps.getPlatformReplyLocale, channelRegistry: deps.channelRegistry, retryEngine: deps.retryEngine, deliveryQueue: deps.deliveryQueue, From e8dd9cf352292fae3e1ba014bb30241117182189 Mon Sep 17 00:00:00 2001 From: comis-agent Date: Mon, 27 Jul 2026 09:37:42 +0300 Subject: [PATCH 10/33] fix(mcp): enforce the call deadline, keep tool-schema constraints, stop the false "running" MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four defects found by replaying an operator's real message sequence against a live MCP server. The heavy report turn they all landed on went from a 603s wall-clock timeout to a 24s honest answer. **The call deadline was not a deadline.** `resetTimeoutOnProgress` restarts the timeout on every progress notification and the SDK applies no ceiling unless `maxTotalTimeout` is passed ("If not specified, there is no maximum total timeout"). So `integrations.mcp.callToolTimeoutMs` silently degraded into a per-progress-GAP timeout while the config docs, the key's name, and the expiry hint's promise that an identical retry "re-expires the same deadline" all called it the total. Ground truth: a 120000ms cap with recorded call durations of 200877ms and 106246ms, and one call free to consume an entire turn budget. Passing `maxTotalTimeout` makes the configured value mean what it says. **Every JSON-Schema constraint was stripped from MCP tool schemas.** `jsonSchemaToTypeBox` mapped `type:"number"` to a bare `Type.Number()`, dropping `minimum`/`maximum`/`enum`/`pattern`/`description`/`default` — and that converted schema IS the model-facing `parameters`. The model could not see bounds the server had published, so it guessed and took an MCP -32602 rejection per guess: four of eight tool failures in one live turn were exactly this (`page_size` below a minimum of 10, `top_n` at 200 against a maximum of 100, an out-of-range enum). Constraint keywords are now carried through verbatim, including for a typeless `{enum:[...]}` that previously collapsed to Any. **A finished frame kept claiming to be running.** The "(running N s)" suffix was appended whenever the caller supplied `elapsedMs`, with no reference to whether anything was in flight — so a failed tool rendered "❌ … (running 0 s)", an outcome and a running claim on one line, and a finished turn left a pill reading "(running 475 s)" ticking after the work was over. Gated on an event actually being in flight; an empty frame still opens with (running 0 s). **A config parse error crash-looped with no reason.** The parser's own message — which carries the line and column — went only into `details`, while `message` is the single string the boot FATAL prints. A broken edit produced "Failed to parse config file: " and nothing else, on a systemd restart loop that repeated it 40 times. The reason now rides the message; only its first line is taken, so a YAML source excerpt never reaches a log. --- docs/reference/config-yaml.mdx | 2 +- .../src/shared/strategies/render.test.ts | 54 ++++++++++++++ .../channels/src/shared/strategies/render.ts | 26 ++++++- packages/core/src/config/loader.test.ts | 25 +++++++ packages/core/src/config/loader.ts | 25 ++++++- .../src/skills/bridge/mcp-tool-bridge.test.ts | 70 +++++++++++++++++++ .../src/skills/bridge/mcp-tool-bridge.ts | 57 ++++++++++++--- .../mcp-client/mcp-client-call.test.ts | 25 +++++++ .../mcp-client/mcp-client-call.ts | 10 +++ 9 files changed, 283 insertions(+), 11 deletions(-) diff --git a/docs/reference/config-yaml.mdx b/docs/reference/config-yaml.mdx index 28ad3a3d11..d0cd66cbb2 100644 --- a/docs/reference/config-yaml.mdx +++ b/docs/reference/config-yaml.mdx @@ -1880,7 +1880,7 @@ External service integrations for search, MCP servers, media processing, and aut | Key | Type | Default | Description | |-----|------|---------|-------------| -| `callToolTimeoutMs` | `number` | `120000` | Default timeout for MCP tool calls in ms. On expiry the tool error **names this key and its current value** and states that an identical retry re-expires the same deadline, so the caller narrows the request (smaller page / date window / fewer entities) instead. This key is an **immutable config path** — an agent cannot patch it via the `gateway` tool, so the error says so explicitly and directs the change to an operator. Raising it means editing this file and restarting the daemon. Slow report and media tools may need well over two minutes. | +| `callToolTimeoutMs` | `number` | `120000` | Default timeout for MCP tool calls in ms. On expiry the tool error **names this key and its current value** and states that an identical retry re-expires the same deadline, so the caller narrows the request (smaller page / date window / fewer entities) instead. This key is an **immutable config path** — an agent cannot patch it via the `gateway` tool, so the error says so explicitly and directs the change to an operator. Raising it means editing this file and restarting the daemon. Slow report and media tools may need well over two minutes. This is an **absolute** ceiling: a server that streams progress notifications resets the idle portion of the deadline but can never exceed this total, so one chatty tool call cannot consume a whole turn's budget. | | `servers` | `McpServerEntry[]` | `[]` | List of MCP servers | **MCP Server Entry (integrations.mcp.servers[])** diff --git a/packages/channels/src/shared/strategies/render.test.ts b/packages/channels/src/shared/strategies/render.test.ts index fccba0bfc2..b0c246a31d 100644 --- a/packages/channels/src/shared/strategies/render.test.ts +++ b/packages/channels/src/shared/strategies/render.test.ts @@ -313,6 +313,60 @@ describe("renderFrameText", () => { // events (so the running flow stays calm and the closing ✓ done is the only // success marker); the running-marker presence is the companion contract. // The ✓-absence assertion is preserved below. + // The "(running N s)" suffix was appended whenever the caller supplied + // `elapsedMs`, with no reference to whether anything was actually in flight. + // So a frame whose every event had TERMINATED still claimed to be running. + // Live: a failed tool rendered "❌ using · (running 0 s)" — + // an outcome and a running claim on one line — and a finished turn left a + // pill reading "(running 475 s)" that outlived the turn it described. + describe("the running suffix tracks in-flight work, not wall-clock", () => { + it("omits the suffix when every visible event has terminated", () => { + const out = renderFrameText( + frame({ + visibleEvents: [ + event({ kind: "tool", phase: "end", status: "failed", toolName: "events_stats" }), + ], + }), + DEFAULT_THEME_MARKERS, + 0, + ); + expect(out).not.toContain("(running"); + }); + + it("omits the suffix for a completed event even after a long elapsed time", () => { + const out = renderFrameText( + frame({ + visibleEvents: [ + event({ kind: "tool", phase: "end", status: "completed", toolName: "vehicles_list" }), + ], + }), + DEFAULT_THEME_MARKERS, + 475_000, + ); + expect(out).not.toContain("(running"); + }); + + it("keeps the suffix while at least one event is still in flight", () => { + const out = renderFrameText( + frame({ + visibleEvents: [ + event({ kind: "tool", phase: "end", status: "completed", toolName: "vehicles_list" }), + event({ kind: "tool", phase: "start", status: "running", toolName: "trips_stats" }), + ], + }), + DEFAULT_THEME_MARKERS, + 12_000, + ); + expect(out).toContain("(running 12 s)"); + }); + + it("keeps the opening (running 0 s) on a frame with no events yet", () => { + // The strategies send an opening placeholder BEFORE the first event + // lands; an empty frame is in-flight by construction, not finished. + expect(renderFrameText(frame(), DEFAULT_THEME_MARKERS, 0)).toContain("(running 0 s)"); + }); + }); + it("does NOT prefix the success ✓ on a kept completed end event (no per-step ✓ during running phase)", () => { const completedFrame = frame({ visibleEvents: [ diff --git a/packages/channels/src/shared/strategies/render.ts b/packages/channels/src/shared/strategies/render.ts index 9d4dea5254..bc4660d0dc 100644 --- a/packages/channels/src/shared/strategies/render.ts +++ b/packages/channels/src/shared/strategies/render.ts @@ -137,6 +137,18 @@ export function eventLabel(event: ActivityEvent, markers?: ActivityStatusMarkers * here. LinePerEvent and DigestOnly do NOT call this function — they use * `eventLabel(event)` per-event and own their own elapsed display. */ +/** + * True while this event still represents work in progress. + * + * Both fields are checked because they answer slightly different questions and + * either one alone leaves a hole: `phase === "end"` marks the event that closes + * a call, while `status` names the outcome. A terminal status on a + * non-"end" phase (a skipped step) is finished too. + */ +function isInFlight(event: ActivityEvent): boolean { + return event.phase !== "end" && event.status === "running"; +} + export function renderFrameText( frame: ActivityRenderFrame, markers?: ActivityStatusMarkers, @@ -204,7 +216,19 @@ export function renderFrameText( // so `elapsedMs === 0` legitimately produces `(running 0 s)` on the first // apply()) AND `frame.planSnapshot === undefined` (the plan header above // already conveys progress — no double-display). - if (frame.planSnapshot === undefined && elapsedMs !== undefined) { + // + // Gated on something ACTUALLY being in flight. `elapsedMs !== undefined` + // alone made the suffix a wall-clock readout that contradicted the very + // lines above it: a frame whose only event was a FAILED tool rendered + // "❌ … (running 0 s)" — an outcome and a running claim together — and a + // finished turn left "(running 475 s)" ticking after the work was over. An + // event whose phase is "end" (or whose status is terminal) is done; when + // every visible event is done, nothing is running and the suffix is a lie. + // An EMPTY frame stays in-flight: the strategies paint an opening + // placeholder before the first event lands. + const somethingInFlight = frame.visibleEvents.length === 0 + || frame.visibleEvents.some(isInFlight); + if (frame.planSnapshot === undefined && elapsedMs !== undefined && somethingInFlight) { const seconds = Math.floor(elapsedMs / 1000); lines.push(`(running ${seconds} s)`); } diff --git a/packages/core/src/config/loader.test.ts b/packages/core/src/config/loader.test.ts index 8046d790bb..8b815945cc 100644 --- a/packages/core/src/config/loader.test.ts +++ b/packages/core/src/config/loader.test.ts @@ -92,6 +92,31 @@ agents: } }); + // The parser's own reason (with its line/column) landed only in `details`, + // while `message` — the single string the boot FATAL prints — named just + // the file. An operator whose edit broke the YAML got + // "FATAL: Bootstrap failed: Failed to parse config file: ", no + // location, and a systemd restart loop repeating it. Live: 40 restarts on + // one stray newline inside a quoted scalar. + it("names the parser's reason and location in the message, not only in details", () => { + const dir = makeTmpDir(); + // A real-world break: an unescaped newline inside a double-quoted scalar. + const filePath = writeFile(dir, "broken.yaml", 'agents:\n default:\n name: "un\nterminated\n'); + + const result = loadConfigFile(filePath); + expect(result.ok).toBe(false); + if (!result.ok) { + expect(result.error.code).toBe("PARSE_ERROR"); + // The file still has to be named — that part was never wrong. + expect(result.error.message).toContain("broken.yaml"); + // ...and now the reason rides along, so the message alone is actionable. + expect(result.error.message.length).toBeGreaterThan( + `Failed to parse config file: ${filePath}`.length, + ); + expect(result.error.message).toMatch(/line \d+/i); + } + }); + it("returns empty object for empty file", () => { const dir = makeTmpDir(); const filePath = writeFile(dir, "empty.yaml", ""); diff --git a/packages/core/src/config/loader.ts b/packages/core/src/config/loader.ts index b3dbfab72c..6d78ce41d6 100644 --- a/packages/core/src/config/loader.ts +++ b/packages/core/src/config/loader.ts @@ -113,15 +113,38 @@ export function loadConfigFile( return ok(processed); } catch (e) { + // Carry the parser's OWN reason into the message. It used to live only in + // `details`, and the boot FATAL prints `message` alone — so a broken edit + // produced "Failed to parse config file: " with no line, no column, + // no reason, on a systemd restart loop that repeated it indefinitely. The + // YAML parser already knows exactly what it choked on and where; not + // passing that through is throwing away the answer at the one moment the + // operator has nothing else to go on (the daemon never reaches its logs). + const reason = parseFailureReason(e); return err({ code: "PARSE_ERROR", - message: `Failed to parse config file: ${resolved}`, + message: reason === undefined + ? `Failed to parse config file: ${resolved}` + : `Failed to parse config file: ${resolved} — ${reason}`, path: resolved, details: e, }); } } +/** + * The underlying parser's single-line reason, trimmed for a boot-FATAL line. + * + * YAML errors carry a multi-line body (message + a source excerpt + a caret). + * The excerpt can echo config CONTENT, which must not reach a log line, so only + * the first line — the reason and its `at line N, column M` — is taken. + */ +function parseFailureReason(e: unknown): string | undefined { + if (!(e instanceof Error) || e.message.length === 0) return undefined; + const firstLine = e.message.split("\n")[0]!.trim(); + return firstLine.length === 0 ? undefined : firstLine.slice(0, 300); +} + /** * Validate a raw config object against the AppConfigSchema. * diff --git a/packages/skills/src/skills/bridge/mcp-tool-bridge.test.ts b/packages/skills/src/skills/bridge/mcp-tool-bridge.test.ts index ed23198b0f..c65b34a791 100644 --- a/packages/skills/src/skills/bridge/mcp-tool-bridge.test.ts +++ b/packages/skills/src/skills/bridge/mcp-tool-bridge.test.ts @@ -45,6 +45,76 @@ function makeCallTool(): McpClientManager["callTool"] { // jsonSchemaToTypeBox // --------------------------------------------------------------------------- +// The converted schema IS the model-facing `parameters` for every MCP tool +// (mcp-tool-bridge.ts: `parameters: typeboxSchema`), so a keyword dropped here +// is a constraint the model never sees. Live: four of eight tool failures in +// one turn were MCP `-32602` rejections — `page_size` below a minimum of 10, +// `top_n` above a maximum of 100, an out-of-range enum — each a constraint the +// server declared and the conversion discarded. The model was guessing at +// bounds that were published all along, and each guess cost a turn step and +// painted a failure at the user. +describe("jsonSchemaToTypeBox preserves the constraints the model needs", () => { + it("keeps numeric bounds", () => { + const s = jsonSchemaToTypeBox({ type: "number", minimum: 10, maximum: 100 }) as Record; + expect(s.minimum).toBe(10); + expect(s.maximum).toBe(100); + }); + + it("keeps integer bounds and exclusive variants", () => { + const s = jsonSchemaToTypeBox({ + type: "integer", minimum: 1, exclusiveMaximum: 50, multipleOf: 5, + }) as Record; + expect(s.minimum).toBe(1); + expect(s.exclusiveMaximum).toBe(50); + expect(s.multipleOf).toBe(5); + }); + + it("keeps string enum, pattern, format and length bounds", () => { + const s = jsonSchemaToTypeBox({ + type: "string", enum: ["pto", "scan"], pattern: "^[a-z]+$", format: "date", minLength: 2, maxLength: 8, + }) as Record; + expect(s.enum).toEqual(["pto", "scan"]); + expect(s.pattern).toBe("^[a-z]+$"); + expect(s.format).toBe("date"); + expect(s.minLength).toBe(2); + expect(s.maxLength).toBe(8); + }); + + it("keeps description and default so the model reads the tool's own guidance", () => { + const s = jsonSchemaToTypeBox({ + type: "number", description: "rows per page", default: 25, + }) as Record; + expect(s.description).toBe("rows per page"); + expect(s.default).toBe(25); + }); + + it("keeps array bounds and item constraints", () => { + const s = jsonSchemaToTypeBox({ + type: "array", minItems: 1, maxItems: 5, items: { type: "string", enum: ["a", "b"] }, + }) as Record; + expect(s.minItems).toBe(1); + expect(s.maxItems).toBe(5); + expect((s.items as Record).enum).toEqual(["a", "b"]); + }); + + it("keeps nested property constraints through an object", () => { + const s = jsonSchemaToTypeBox({ + type: "object", + properties: { page_size: { type: "number", minimum: 10 } }, + required: ["page_size"], + }) as Record; + const props = s.properties as Record>; + expect(props.page_size.minimum).toBe(10); + }); + + it("keeps an enum that declares no type at all", () => { + // A bare `{enum:[...]}` has no `type`, so it fell to the Type.Any() + // fallback and the allowed values vanished. + const s = jsonSchemaToTypeBox({ enum: ["DrivePermissionGroup", "VehicleGroup"] }) as Record; + expect(s.enum).toEqual(["DrivePermissionGroup", "VehicleGroup"]); + }); +}); + describe("jsonSchemaToTypeBox", () => { it("converts string type", () => { const result = jsonSchemaToTypeBox({ type: "string" }); diff --git a/packages/skills/src/skills/bridge/mcp-tool-bridge.ts b/packages/skills/src/skills/bridge/mcp-tool-bridge.ts index a7d6d98879..5f4c4faa71 100644 --- a/packages/skills/src/skills/bridge/mcp-tool-bridge.ts +++ b/packages/skills/src/skills/bridge/mcp-tool-bridge.ts @@ -68,29 +68,30 @@ export function classifyMcpErrorType(errorText: string | undefined): string { */ export function jsonSchemaToTypeBox(schema: Record): TSchema { const type = schema.type; + const annotations = carriedKeywords(schema); if (type === "string") { - return Type.String(); + return Type.String(annotations); } if (type === "number") { - return Type.Number(); + return Type.Number(annotations); } if (type === "integer") { - return Type.Integer(); + return Type.Integer(annotations); } if (type === "boolean") { - return Type.Boolean(); + return Type.Boolean(annotations); } if (type === "array") { const items = schema.items as Record | undefined; if (items) { - return Type.Array(jsonSchemaToTypeBox(items)); + return Type.Array(jsonSchemaToTypeBox(items), annotations); } - return Type.Array(Type.Any()); + return Type.Array(Type.Any(), annotations); } if (type === "object") { @@ -98,7 +99,7 @@ export function jsonSchemaToTypeBox(schema: Record): TSchema { const required = (schema.required as string[]) ?? []; if (!properties) { - return Type.Object({}); + return Type.Object({}, annotations); } const typeboxProps: Record = {}; @@ -107,13 +108,53 @@ export function jsonSchemaToTypeBox(schema: Record): TSchema { typeboxProps[key] = required.includes(key) ? converted : Type.Optional(converted); } - return Type.Object(typeboxProps); + return Type.Object(typeboxProps, annotations); + } + + // A typeless `{enum:[...]}` is legal JSON Schema and common in MCP servers. + // Carrying the keywords here keeps the allowed values instead of erasing + // them into an untyped Any. + if (Object.keys(annotations).length > 0) { + return Type.Unsafe(annotations); } // Fallback for unknown or complex schema types return Type.Any(); } +/** + * JSON Schema keywords copied through the conversion verbatim. + * + * Deliberately a copy-through list, not an interpretation: TypeBox stores + * unknown options as plain JSON Schema keywords, so a keyword listed here + * survives into the tool's published `parameters` untouched. `type`, + * `properties`, `items` and `required` are excluded because the conversion + * above reconstructs them structurally. + */ +const CARRIED_SCHEMA_KEYWORDS = [ + // Numeric bounds. Their loss is what made a model guess `page_size` and + // `top_n` and take an MCP -32602 rejection for each guess. + "minimum", "maximum", "exclusiveMinimum", "exclusiveMaximum", "multipleOf", + // Value sets — an out-of-range enum is unguessable without them. + "enum", "const", + // String shape. + "minLength", "maxLength", "pattern", "format", + // Array/object cardinality. + "minItems", "maxItems", "uniqueItems", "minProperties", "maxProperties", + // The tool's own prose and defaults: the model reads these as guidance, and + // dropping them silently discarded the server's advice about its own inputs. + "description", "title", "default", "examples", +] as const; + +/** Pick the carried keywords present on a source schema. */ +function carriedKeywords(schema: Record): Record { + const carried: Record = {}; + for (const keyword of CARRIED_SCHEMA_KEYWORDS) { + if (schema[keyword] !== undefined) carried[keyword] = schema[keyword]; + } + return carried; +} + // --------------------------------------------------------------------------- // Description truncation // --------------------------------------------------------------------------- diff --git a/packages/skills/src/skills/integrations/mcp-client/mcp-client-call.test.ts b/packages/skills/src/skills/integrations/mcp-client/mcp-client-call.test.ts index 5f59ea0c35..c38cb961b9 100644 --- a/packages/skills/src/skills/integrations/mcp-client/mcp-client-call.test.ts +++ b/packages/skills/src/skills/integrations/mcp-client/mcp-client-call.test.ts @@ -290,7 +290,32 @@ describe("request correlation", () => { timeout: 5000, onprogress: expect.any(Function), resetTimeoutOnProgress: true, + maxTotalTimeout: 5000, }, ); }); + + // `resetTimeoutOnProgress` restarts the timeout on EVERY progress + // notification, and the SDK applies no total ceiling unless `maxTotalTimeout` + // is passed ("If not specified, there is no maximum total timeout"). So a + // server that emits progress held a call open indefinitely while the config + // key, the docs, and the expiry hint all called it the call deadline. Live: + // a 120000ms cap with observed call durations of 139478ms and 110004ms, and + // single calls free to consume the whole turn budget. + it("bounds a progress-emitting call with an absolute ceiling, not just a per-gap timeout", async () => { + const serverName = "inventory"; + const state = makeConnectedState(serverName, () => + Promise.resolve({ content: [{ type: "text", text: "{}" }] }), + ); + const deps = { logger: makeLogger() } as unknown as McpClientManagerDeps; + + await runWithContext(makeContext(), () => + callTool(state, deps, `mcp:${serverName}/inventory_items_list`, {}), + ); + + const opts = (state.connections.get(serverName)?.client.callTool as ReturnType) + .mock.calls[0]![2] as Record; + expect(opts.resetTimeoutOnProgress).toBe(true); + expect(opts.maxTotalTimeout).toBe(state.options.callToolTimeoutMs); + }); }); diff --git a/packages/skills/src/skills/integrations/mcp-client/mcp-client-call.ts b/packages/skills/src/skills/integrations/mcp-client/mcp-client-call.ts index 2f63070e08..509fa4dc0d 100644 --- a/packages/skills/src/skills/integrations/mcp-client/mcp-client-call.ts +++ b/packages/skills/src/skills/integrations/mcp-client/mcp-client-call.ts @@ -209,10 +209,20 @@ export async function callTool( undefined, { timeout: state.options.callToolTimeoutMs, + // `maxTotalTimeout` rides along with the progress reset, ALWAYS. + // `resetTimeoutOnProgress` restarts `timeout` on every progress + // notification, and the SDK applies no ceiling of its own ("If not + // specified, there is no maximum total timeout") — so without this the + // configured deadline silently degrades into a per-progress-GAP + // timeout, and a chatty server can hold one call open for the whole + // turn. That contradicts the key's documented meaning and the expiry + // hint's promise that an identical retry "re-expires the same + // deadline". Live: a 120000ms cap with 139478ms and 110004ms calls. ...(requestTraceId ? { onprogress: () => {}, resetTimeoutOnProgress: true, + maxTotalTimeout: state.options.callToolTimeoutMs, } : {}), }, From 9cb2bb9e68029c591bcb20edb2d43703b8875484 Mon Sep 17 00:00:00 2001 From: comis-agent Date: Mon, 27 Jul 2026 10:01:35 +0300 Subject: [PATCH 11/33] fix(obs): stop classifying a breaker refusal as a timeout, and fix the timeout advice MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two observability defects from the same live incident as 2f1fd8b92. **A breaker refusal was reported as a 3ms timeout.** `buildBlockReason` quotes the original failure verbatim (`…with the same error: "…timed out…"`), and the bridge fed that text to the MCP error classifier, which read the QUOTATION as a fresh timeout. The blocked call was then published as `tool.timeout {timeoutMs: 3}` and `errorKind:"timeout"` — 3ms being how long the breaker took to refuse, presented as an expired 120000ms deadline. That is a false report at the exact surface an operator reads to find a real one, and it inflates the timeout population every timeout heuristic keys on. A blocked call never reached the server, so it has no transport outcome to classify at all. It is now recognized before the classifier and recorded as `precondition` / `runtime_guard` / `transportOk:false`, so no `tool:timeout` is emitted for it. The recognizer lives with the producer (`isBreakerBlockMessage`) and requires BOTH structural invariants `buildBlockReason` guarantees, so a tool that merely echoes one phrase is not mistaken for a block. **`provider_timeout` led its advice with the one step that makes things worse.** "raise the per-call timeout" named no knob, and a turn's wall-clock budget is fixed — a longer per-call deadline just buys a longer burn before the same abort. Narrowing now leads. The deadline is still named, but by its exact key and with the fact that it is an immutable path an agent cannot patch, so the reader knows the step needs an operator and a daemon restart. --- packages/agent/src/bridge/pi-event-bridge.ts | 14 +++++++ .../src/safety/tool-retry-breaker.test.ts | 40 ++++++++++++++++++- .../agent/src/safety/tool-retry-breaker.ts | 22 ++++++++++ .../obs-explain-heuristics.test.ts | 29 ++++++++++++++ .../obs-handlers/obs-explain-heuristics.ts | 13 +++++- 5 files changed, 116 insertions(+), 2 deletions(-) diff --git a/packages/agent/src/bridge/pi-event-bridge.ts b/packages/agent/src/bridge/pi-event-bridge.ts index 1829fa06d0..bcaa3081f0 100644 --- a/packages/agent/src/bridge/pi-event-bridge.ts +++ b/packages/agent/src/bridge/pi-event-bridge.ts @@ -12,6 +12,7 @@ import { shouldCompact } from "@earendil-works/pi-coding-agent"; import { isRelayedBackgroundFailure } from "../safety/background-failure-attribution.js"; +import { isBreakerBlockMessage } from "../safety/tool-retry-breaker.js"; import type { AgentSessionEvent } from "@earendil-works/pi-coding-agent"; import type { AssistantMessage } from "@earendil-works/pi-ai"; import { @@ -1135,6 +1136,19 @@ export function createPiEventBridge(deps: PiEventBridgeDeps): PiEventBridgeResul // The runtime blocked the call before the tool or MCP transport // boundary, so no external dependency was contacted. transportOk = false; + } else if (isBreakerBlockMessage(errorText)) { + // The retry breaker REFUSED this call — it never ran, so there + // is no transport outcome to classify. The block text quotes the + // original failure verbatim, so letting it fall through to the + // MCP classifier below re-read that quotation as a fresh + // failure of the quoted kind: a blocked call came back + // errorKind:"timeout" and emitted `tool.timeout {timeoutMs: 3}`, + // publishing the breaker's 3ms refusal latency as an expired + // 120000ms deadline. "precondition" is the honest kind — a + // guard the call did not satisfy. + toolErrorKind = "precondition"; + classifiedFailureBy = "runtime_guard"; + transportOk = false; } else if (mcpServer !== undefined) { const mcpKind = classifyMcpErrorType(errorText); toolErrorKind = mcpKind === "timeout" diff --git a/packages/agent/src/safety/tool-retry-breaker.test.ts b/packages/agent/src/safety/tool-retry-breaker.test.ts index 86bcc58ba4..12acfb4faf 100644 --- a/packages/agent/src/safety/tool-retry-breaker.test.ts +++ b/packages/agent/src/safety/tool-retry-breaker.test.ts @@ -1,7 +1,9 @@ // SPDX-License-Identifier: Apache-2.0 import { describe, it, expect } from "vitest"; import { wrapExternalContent } from "@comis/core"; -import { createToolRetryBreaker, extractErrorTag, buildBlockReason } from "./tool-retry-breaker.js"; +import { createToolRetryBreaker, extractErrorTag, buildBlockReason, + isBreakerBlockMessage, +} from "./tool-retry-breaker.js"; import type { ToolRetryBreaker } from "./tool-retry-breaker.js"; describe("tool retry breaker", () => { @@ -1239,3 +1241,39 @@ describe("tool retry breaker", () => { }); }); }); + +// A breaker BLOCK is a Comis runtime message, not a transport response. It +// quotes the original failure verbatim ("...with the same error: \"...timed +// out...\""), so downstream classifiers that sniff error text re-read the +// quotation as a fresh failure of that kind. Live: a blocked MCP call was +// classified errorKind:"timeout" and emitted `tool.timeout {timeoutMs: 3}` — +// 3ms being how long the breaker took to say no, presented as an expired +// deadline. The producer owns the recognizer so consumers can tell the two +// apart before classifying. +describe("isBreakerBlockMessage", () => { + it("recognizes a tool-level block that quotes a timeout", () => { + const reason = buildBlockReason( + "mcp__vendor--report", 13, + 'MCP tool error: MCP tool "report" on server "vendor" timed out — it exceeded the call deadline of 120000ms', + [], "timeout", true, + ); + expect(isBreakerBlockMessage(reason)).toBe(true); + }); + + it("recognizes a parameter-validation block", () => { + const reason = buildBlockReason("some_tool", 3, "bad args", [], "invalid_params", false); + expect(isBreakerBlockMessage(reason)).toBe(true); + }); + + it("does NOT match the underlying error the block quotes", () => { + expect(isBreakerBlockMessage( + 'MCP tool error: MCP tool "report" on server "vendor" timed out — it exceeded the call deadline of 120000ms', + )).toBe(false); + }); + + it("does NOT match arbitrary tool output or an empty string", () => { + expect(isBreakerBlockMessage("DO NOT retry this tool.")).toBe(false); + expect(isBreakerBlockMessage("")).toBe(false); + expect(isBreakerBlockMessage(undefined)).toBe(false); + }); +}); diff --git a/packages/agent/src/safety/tool-retry-breaker.ts b/packages/agent/src/safety/tool-retry-breaker.ts index b167060c53..d1dbe05f30 100644 --- a/packages/agent/src/safety/tool-retry-breaker.ts +++ b/packages/agent/src/safety/tool-retry-breaker.ts @@ -391,6 +391,28 @@ export function buildBlockReason( return full.slice(0, 500); } +/** + * True when this text is a breaker BLOCK message rather than a tool's own error. + * + * Load-bearing for classification. A block quotes the original failure verbatim + * (`…with the same error: "…timed out…"`), so any consumer that classifies by + * sniffing error text reads the QUOTATION as a fresh failure of that kind. Live: + * a blocked MCP call was classified `errorKind:"timeout"` and emitted + * `tool.timeout {timeoutMs: 3}` — 3ms being how long the breaker took to refuse, + * published as an expired deadline. A blocked call never reached the server; it + * has no transport outcome to classify at all. + * + * Matched on the two structural invariants {@link buildBlockReason} guarantees: + * one of its two header forms, plus its refusal block. Both are required, so a + * tool that merely echoes one phrase is not mistaken for a block. + */ +export function isBreakerBlockMessage(text: string | undefined): boolean { + if (text === undefined || text.length === 0) return false; + if (!text.includes("DO NOT retry this tool. Instead:")) return false; + return /Tool "[^"]+" (?:has failed \d+ (?:total|consecutive) times|failed parameter validation \d+ times)/ + .test(text); +} + // --------------------------------------------------------------------------- // Factory // --------------------------------------------------------------------------- diff --git a/packages/daemon/src/api/obs-handlers/obs-explain-heuristics.test.ts b/packages/daemon/src/api/obs-handlers/obs-explain-heuristics.test.ts index 45740e5944..51575ac0cc 100644 --- a/packages/daemon/src/api/obs-handlers/obs-explain-heuristics.test.ts +++ b/packages/daemon/src/api/obs-handlers/obs-explain-heuristics.test.ts @@ -201,6 +201,35 @@ describe("obs-explain-heuristics", () => { expect(r!.suggestedNextSteps.length).toBeGreaterThan(0); }); + // The first suggested step was "raise the per-call timeout", which names no + // knob and leads with the one action that makes a deadline-bound turn worse: + // a longer deadline buys a longer burn against the same fixed turn budget. + // Narrowing the request is the move that actually completes, and the MCP + // deadline lives at a config path an agent cannot patch — so the advice has + // to name it and say who can change it. + it("insurance: provider_timeout leads with narrowing and names the MCP deadline knob", () => { + const r = rootCause( + makeSignals({ + failures: [ + { + seq: 0, + toolName: "mcp__vendor--report", + classifiedFailureBy: "mcp_classifier", + transportOk: true, + errorKind: "timeout", + resultDigest: "abc", + resultBytes: 10, + errorPreview: "timed out — it exceeded the call deadline of 120000ms", + }, + ], + }), + ); + expect(r!.code).toBe("provider_timeout"); + expect(r!.suggestedNextSteps[0]).toMatch(/narrow|smaller|fewer/i); + expect(r!.suggestedNextSteps.join(" ")).toContain("integrations.mcp.callToolTimeoutMs"); + expect(r!.suggestedNextSteps.join(" ")).toMatch(/operator/i); + }); + it("insurance: context_bloat (offloads ≥ N + token spike)", () => { const r = rootCause( makeSignals({ diff --git a/packages/daemon/src/api/obs-handlers/obs-explain-heuristics.ts b/packages/daemon/src/api/obs-handlers/obs-explain-heuristics.ts index 83d18e90ea..5207f38eb7 100644 --- a/packages/daemon/src/api/obs-handlers/obs-explain-heuristics.ts +++ b/packages/daemon/src/api/obs-handlers/obs-explain-heuristics.ts @@ -290,9 +290,20 @@ export const HEURISTICS: ReadonlyArray<(s: IncidentSignals) => RootCause | null> "provider timeout: " + failure.toolName + " exceeded its deadline (errorKind=timeout)", + // Narrowing FIRST. "Raise the per-call timeout" led this list, which is + // the one action that makes a deadline-bound turn worse: the turn budget + // is fixed, so a longer per-call deadline just buys a longer burn before + // the same wall-clock abort. Narrowing is what actually completes. The + // deadline is still worth naming — but by its exact key, and with the + // fact that an agent cannot patch it (immutable config path), so the + // reader knows the step needs an operator and a daemon restart. suggestedNextSteps: [ - "raise the per-call timeout or reduce the request size for " + failure.toolName, + "narrow the request for " + failure.toolName + + " (a smaller page / date window / fewer entities) so it completes inside the deadline", "check provider latency / rate-limit headroom", + "if it genuinely needs longer, an operator can raise" + + " `integrations.mcp.callToolTimeoutMs` (MCP tools) in the config file" + + " and restart the daemon — it is an immutable path an agent cannot patch", "obs.explain depth=full", ], }; From 21a6771f70205734defe2456f94227917345b750 Mon Sep 17 00:00:00 2001 From: comis-agent Date: Mon, 27 Jul 2026 10:36:47 +0300 Subject: [PATCH 12/33] fix(mcp): make the call ceiling unconditional; make a breaker refusal authoritative MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two corrections to 2f1fd8b92 and c777c5501, both found by re-verifying them against the live box instead of trusting the unit tests. **The absolute ceiling was scoped to the tracing branch.** `maxTotalTimeout` was added inside the `requestTraceId` conditional next to `resetTimeoutOnProgress`, so it applied only when a trace context happened to exist — and the untraced paths, which is exactly where a long-running background call runs, kept no ceiling at all. A deadline enforced only when tracing is on is not a deadline. It is now passed unconditionally, alongside (not inside) the progress reset. The two exact-equality assertions on the SDK options object are updated: the ceiling being present on every call IS the contract. **A breaker refusal now overrides rather than falls back.** The first cut gated it on `toolErrorKind === undefined`, which makes the classification depend on whether some upstream detector had already assigned a kind by sniffing the same quoted text. A refusal is authoritative — the call never ran, so nothing in the text is evidence of a transport outcome — and it must not be order-dependent. The bridge test locks the contract: a refusal quoting a timeout is `precondition` and emits no `tool:timeout`. Honesty note on the second one: I have not observed the ordering actually flip in production, and the regression test passes against the previous code too. This is defense-in-depth on a real semantic error, not a fix for a reproduced failure. --- .../agent/src/bridge/pi-event-bridge.test.ts | 32 +++++++++++++++++++ packages/agent/src/bridge/pi-event-bridge.ts | 32 +++++++++++-------- .../skills/integrations/mcp-client.test.ts | 11 +++++-- .../mcp-client/mcp-client-call.test.ts | 19 +++++++++++ .../mcp-client/mcp-client-call.ts | 24 ++++++++------ 5 files changed, 91 insertions(+), 27 deletions(-) diff --git a/packages/agent/src/bridge/pi-event-bridge.test.ts b/packages/agent/src/bridge/pi-event-bridge.test.ts index 187d5036ce..f83c289ef8 100644 --- a/packages/agent/src/bridge/pi-event-bridge.test.ts +++ b/packages/agent/src/bridge/pi-event-bridge.test.ts @@ -866,6 +866,38 @@ describe("createPiEventBridge", () => { expect(endEmit![1].errorKind).toBe("validation"); }); + // The breaker's block message QUOTES the original failure verbatim, so every + // text-sniffing classifier upstream reads the quotation and reports a fresh + // failure of the quoted kind. Live: a blocked MCP call came back + // errorKind:"timeout" and emitted `tool.timeout {timeoutMs: 3}` — the + // breaker's 3ms refusal latency published as an expired 120000ms deadline. + // The refusal must OVERRIDE, not merely fall back: a first attempt gated on + // "no kind assigned yet" never fired, because the failure detector had + // already set one. + it("a breaker refusal quoting a timeout is precondition, and emits no tool:timeout", () => { + const { listener } = createPiEventBridge(deps); + + const result = { + message: + 'Tool "mcp__vendor--report" has failed 13 total times with the same error: ' + + '"MCP tool error: MCP tool \"report\" on server \"vendor\" timed out — it ' + + 'exceeded the call deadline of 120000ms". This tool appears to be unavailable.' + + "\n\nDO NOT retry this tool. Instead:\n- Use the data you already have", + }; + listener(makeToolExecutionEndEvent("mcp__vendor--report", "tc-blk", true, result) as any); + + const calls = (deps.eventBus.emit as ReturnType).mock.calls; + const endEmit = calls.find( + (c) => c[0] === "tool:executed" && c[1].toolName === "mcp__vendor--report", + ); + expect(endEmit).toBeDefined(); + expect(endEmit![1].success).toBe(false); + // Not "timeout" — the call never ran, so it has no transport outcome. + expect(endEmit![1].errorKind).toBe("precondition"); + // ...and therefore nothing may be published on the timeout channel. + expect(calls.find((c) => c[0] === "tool:timeout")).toBeUndefined(); + }); + it("isError=true with generic errorText emits errorKind=dependency", () => { const { listener } = createPiEventBridge(deps); diff --git a/packages/agent/src/bridge/pi-event-bridge.ts b/packages/agent/src/bridge/pi-event-bridge.ts index bcaa3081f0..1d6dcd44fe 100644 --- a/packages/agent/src/bridge/pi-event-bridge.ts +++ b/packages/agent/src/bridge/pi-event-bridge.ts @@ -1128,7 +1128,24 @@ export function createPiEventBridge(deps: PiEventBridgeDeps): PiEventBridgeResul // dedicated MCP classifier into the closed ErrorKind union // (timeout → timeout, connection/transport → dependency, // everything else → classifyToolError fallback). - if (toolErrorKind === undefined) { + // A breaker REFUSAL is authoritative and OVERRIDES any earlier + // classification. It cannot be a fallback branch: the block text + // quotes the original failure verbatim (`…with the same error: + // "…timed out…"`), so every text-sniffing classifier upstream — + // the failure detector as well as the MCP classifier — reads that + // QUOTATION and classifies the refusal as a fresh failure of the + // quoted kind. Gating on `toolErrorKind === undefined` therefore + // never fired: the detector had already set "timeout", and the + // refusal was published as `tool.timeout {timeoutMs: 3}` — the + // breaker's 3ms refusal latency dressed up as an expired 120000ms + // deadline. A blocked call never reached the server, so nothing + // about the text is evidence of a transport outcome. + if (!toolSuccess && isBreakerBlockMessage(errorText)) { + // "precondition" — a guard the call did not satisfy. + toolErrorKind = "precondition"; + classifiedFailureBy = "runtime_guard"; + transportOk = false; + } else if (toolErrorKind === undefined) { if (runtimeToolGuard !== undefined) { toolErrorKind = "resource"; classifiedFailureBy = "runtime_guard"; @@ -1136,19 +1153,6 @@ export function createPiEventBridge(deps: PiEventBridgeDeps): PiEventBridgeResul // The runtime blocked the call before the tool or MCP transport // boundary, so no external dependency was contacted. transportOk = false; - } else if (isBreakerBlockMessage(errorText)) { - // The retry breaker REFUSED this call — it never ran, so there - // is no transport outcome to classify. The block text quotes the - // original failure verbatim, so letting it fall through to the - // MCP classifier below re-read that quotation as a fresh - // failure of the quoted kind: a blocked call came back - // errorKind:"timeout" and emitted `tool.timeout {timeoutMs: 3}`, - // publishing the breaker's 3ms refusal latency as an expired - // 120000ms deadline. "precondition" is the honest kind — a - // guard the call did not satisfy. - toolErrorKind = "precondition"; - classifiedFailureBy = "runtime_guard"; - transportOk = false; } else if (mcpServer !== undefined) { const mcpKind = classifyMcpErrorType(errorText); toolErrorKind = mcpKind === "timeout" diff --git a/packages/skills/src/skills/integrations/mcp-client.test.ts b/packages/skills/src/skills/integrations/mcp-client.test.ts index 515b700a4c..b671a3e6ac 100644 --- a/packages/skills/src/skills/integrations/mcp-client.test.ts +++ b/packages/skills/src/skills/integrations/mcp-client.test.ts @@ -611,7 +611,10 @@ describe("McpClientManager", () => { expect(mockCallTool).toHaveBeenCalledWith( { name: "search", arguments: { query: "test" } }, undefined, - { timeout: MCP_CALL_TOOL_TIMEOUT_MS_DEFAULT }, + { + timeout: MCP_CALL_TOOL_TIMEOUT_MS_DEFAULT, + maxTotalTimeout: MCP_CALL_TOOL_TIMEOUT_MS_DEFAULT, + }, ); }); @@ -748,11 +751,13 @@ describe("McpClientManager", () => { await mgr.callTool("mcp:test-server/search", { query: "test" }); - // SDK callTool should receive timeout in third arg (options) + // SDK callTool receives the timeout in the third arg (options), plus the + // absolute ceiling — `maxTotalTimeout` is unconditional, because a + // deadline that applies only on some code paths is not a deadline. expect(mockCallTool).toHaveBeenCalledWith( { name: "search", arguments: { query: "test" } }, undefined, - { timeout: 120_000 }, + { timeout: 120_000, maxTotalTimeout: 120_000 }, ); }); diff --git a/packages/skills/src/skills/integrations/mcp-client/mcp-client-call.test.ts b/packages/skills/src/skills/integrations/mcp-client/mcp-client-call.test.ts index c38cb961b9..344d3dc2c7 100644 --- a/packages/skills/src/skills/integrations/mcp-client/mcp-client-call.test.ts +++ b/packages/skills/src/skills/integrations/mcp-client/mcp-client-call.test.ts @@ -318,4 +318,23 @@ describe("request correlation", () => { expect(opts.resetTimeoutOnProgress).toBe(true); expect(opts.maxTotalTimeout).toBe(state.options.callToolTimeoutMs); }); + + // The ceiling must not be scoped to the tracing branch. Tying it to + // `requestTraceId` left every untraced path — which is where long-running + // background calls run — with no ceiling at all. Live: after a first cut that + // gated it, a background task still recorded 140233ms against a 120000ms cap. + it("applies the absolute ceiling even with no request trace context", async () => { + const serverName = "inventory"; + const state = makeConnectedState(serverName, () => + Promise.resolve({ content: [{ type: "text", text: "{}" }] }), + ); + const deps = { logger: makeLogger() } as unknown as McpClientManagerDeps; + + // No runWithContext → no requestTraceId. + await callTool(state, deps, `mcp:${serverName}/inventory_items_list`, {}); + + const opts = (state.connections.get(serverName)?.client.callTool as ReturnType) + .mock.calls[0]![2] as Record; + expect(opts.maxTotalTimeout).toBe(state.options.callToolTimeoutMs); + }); }); diff --git a/packages/skills/src/skills/integrations/mcp-client/mcp-client-call.ts b/packages/skills/src/skills/integrations/mcp-client/mcp-client-call.ts index 509fa4dc0d..a64eb7d6f5 100644 --- a/packages/skills/src/skills/integrations/mcp-client/mcp-client-call.ts +++ b/packages/skills/src/skills/integrations/mcp-client/mcp-client-call.ts @@ -209,20 +209,24 @@ export async function callTool( undefined, { timeout: state.options.callToolTimeoutMs, - // `maxTotalTimeout` rides along with the progress reset, ALWAYS. - // `resetTimeoutOnProgress` restarts `timeout` on every progress - // notification, and the SDK applies no ceiling of its own ("If not - // specified, there is no maximum total timeout") — so without this the - // configured deadline silently degrades into a per-progress-GAP - // timeout, and a chatty server can hold one call open for the whole - // turn. That contradicts the key's documented meaning and the expiry - // hint's promise that an identical retry "re-expires the same - // deadline". Live: a 120000ms cap with 139478ms and 110004ms calls. + // The absolute ceiling is UNCONDITIONAL — it is not part of the + // tracing branch. `resetTimeoutOnProgress` restarts `timeout` on every + // progress notification and the SDK applies no ceiling of its own + // ("If not specified, there is no maximum total timeout"), so without + // this the configured deadline degrades into a per-progress-GAP + // timeout and a chatty server holds one call open for a whole turn. + // Live: a 120000ms cap with observed 200877ms and 296481ms calls. + // + // Scoping it to `requestTraceId` (as the first cut did) tied the + // deadline's enforcement to whether a trace context happened to + // exist — so the paths WITHOUT one, which is exactly where a + // long-running background call runs, kept no ceiling at all. A + // deadline that applies only when tracing is on is not a deadline. + maxTotalTimeout: state.options.callToolTimeoutMs, ...(requestTraceId ? { onprogress: () => {}, resetTimeoutOnProgress: true, - maxTotalTimeout: state.options.callToolTimeoutMs, } : {}), }, From 1ee3e25d93bb5508553be96efcdb46c525f2f4e9 Mon Sep 17 00:00:00 2001 From: comis-agent Date: Mon, 27 Jul 2026 10:49:42 +0300 Subject: [PATCH 13/33] docs(obs): state that background_task:failed durationMs is the task lifespan, not the call MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Read as a call duration and compared against `integrations.mcp.callToolTimeoutMs`, this field manufactures a phantom deadline breach: the terminal state is committed on a poll, so the span carries launch and polling latency on top of the call. Live, a correctly-capped 120000ms call surfaced here as 138841ms and was twice mistaken for an unenforced deadline — once in a written findings ledger, once in a fix aimed at the wrong layer. The declaration now names the span and points at `tool.result`'s durationMs for the call itself. --- packages/core/src/event-bus/events-infra.ts | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/packages/core/src/event-bus/events-infra.ts b/packages/core/src/event-bus/events-infra.ts index 56d0a96809..e35a3a7130 100644 --- a/packages/core/src/event-bus/events-infra.ts +++ b/packages/core/src/event-bus/events-infra.ts @@ -673,6 +673,19 @@ export interface InfraEvents { taskId: string; toolName: string; error: string; + /** + * The TASK's whole lifespan — promote-time to terminal commit. NOT the + * duration of the underlying tool call. + * + * The gap is real and routinely tens of seconds: the terminal state is + * committed on a POLL, so this span carries launch and polling latency on + * top of the call. Reading it as a call duration and comparing it against + * `integrations.mcp.callToolTimeoutMs` therefore manufactures a phantom + * deadline breach — live, a correctly-capped 120000ms call surfaced here as + * 138841ms and was twice mistaken for an unenforced deadline. For the call's + * own elapsed time use the `tool.result` record's `durationMs`, which is + * scoped to the call itself. + */ durationMs: number; origin: BackgroundTaskOrigin; timestamp: number; From 21034c3274de00ef14b6978e2197362b63d53ad1 Mon Sep 17 00:00:00 2001 From: comis-agent Date: Mon, 27 Jul 2026 11:02:17 +0300 Subject: [PATCH 14/33] fix(agent): localize the tool-failure notice and stop blaming the background poller MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Found by re-running the operator replay end to end. A Hebrew reply came back with `[tool failure] background_tasks reported an error` stapled to it — a bare, bracket-tagged English internal string appended to an otherwise complete answer in another language. Two defects in one line. **It was outside the locale seam.** Every other deterministic platform string now resolves through `LocaleMessageId`; this one was a literal at its append site, so no operator pack could reach it. It is now `tool_failure_notice`. The tool NAME still rides along verbatim — identifiers are never translated (the no-translation principle in docs/operations/multilingual.mdx); only the sentence around it is localized. **It named `background_tasks`.** That is the POLLER, which relays other tools' failures — so the notice pointed the reader at the one tool that was working. The same mis-attribution the retry breaker had in 463c55646, surfacing at the user-facing layer instead of the breaker. The notice now prefers a real failing tool and falls back to a nameless notice rather than blaming the messenger. The two source-grep contract tests that pinned the old literal are updated to pin the new contract: built through the seam, and never naming the poller. --- .../agent/src/executor/degraded-reply-i18n.ts | 19 ++++++- packages/agent/src/executor/degraded-reply.ts | 12 +++++ .../executor/executor-post-execution.test.ts | 21 ++++++-- .../src/executor/executor-post-execution.ts | 51 +++++++++++-------- 4 files changed, 78 insertions(+), 25 deletions(-) diff --git a/packages/agent/src/executor/degraded-reply-i18n.ts b/packages/agent/src/executor/degraded-reply-i18n.ts index 4bd998c30e..eabb191df4 100644 --- a/packages/agent/src/executor/degraded-reply-i18n.ts +++ b/packages/agent/src/executor/degraded-reply-i18n.ts @@ -17,7 +17,8 @@ export type LocaleMessageId = | "advice_fixed_overhead" | "output_starved" | "loop_detected" - | "pipeline_timeout"; + | "pipeline_timeout" + | "tool_failure_notice"; export type LocalePack = Readonly>>; @@ -45,6 +46,9 @@ const ENGLISH_PACK: Readonly> = { "I stopped because I kept repeating an action that wasn't making progress " + "(usually a tool that failed or was blocked) and didn't want to loop. The " + "request may need a different approach, or that capability isn't available here.", + tool_failure_notice: + "\n\nNote: one of the tools I used reported an error, so part of this may be" + + " incomplete — ", pipeline_timeout: "I stopped this request because it was taking too long and hit the time limit " + "for a single turn. Nothing was left half-applied. If it needs many lookups, " @@ -181,6 +185,19 @@ export function selectLoopDetectedReply( * (`executionTimeoutMs`). The model never returned, so there is no partial text * to annotate — this REPLACES the response entirely. */ +/** + * The trailing notice appended when a tool failed and the model's own reply did + * not mention it. The failing tool's NAME is appended verbatim by the caller — + * identifiers stay untranslated in every language (see the no-translation + * principle in docs/operations/multilingual.mdx), only the prose is localized. + */ +export function selectToolFailureNotice( + locale: string | undefined, + catalog: LocaleCatalog = DEFAULT_LOCALE_CATALOG, +): string { + return catalog.resolve(locale, "tool_failure_notice"); +} + export function selectPipelineTimeoutReply( locale: string | undefined, opts: { traceId?: string }, diff --git a/packages/agent/src/executor/degraded-reply.ts b/packages/agent/src/executor/degraded-reply.ts index 93354d4dbd..f819e6239f 100644 --- a/packages/agent/src/executor/degraded-reply.ts +++ b/packages/agent/src/executor/degraded-reply.ts @@ -24,6 +24,7 @@ import { selectContextExhaustedReply, selectLoopDetectedReply, selectPipelineTimeoutReply, + selectToolFailureNotice, type LocaleCatalog, } from "./degraded-reply-i18n.js"; @@ -124,3 +125,14 @@ export function buildPipelineTimeoutReply(opts?: ContextExhaustedReplyOpts): str traceId: opts?.traceId, }, opts?.localeCatalog); } + +/** + * Localized notice that a tool failed, for appending to a reply that did not + * itself mention the failure. The caller appends the tool name verbatim. + */ +export function buildToolFailureNotice( + language?: string, + localeCatalog?: LocaleCatalog, +): string { + return selectToolFailureNotice(language, localeCatalog); +} diff --git a/packages/agent/src/executor/executor-post-execution.test.ts b/packages/agent/src/executor/executor-post-execution.test.ts index a899728679..0e86e0c6e1 100644 --- a/packages/agent/src/executor/executor-post-execution.test.ts +++ b/packages/agent/src/executor/executor-post-execution.test.ts @@ -812,10 +812,23 @@ describe("tool-failure endReason and notice", () => { expect(stripped).toMatch(/function\s+modelAcknowledgedFailure\s*\(/); }); - it("source-grep — failure notice '[tool failure]' appended to result.response at call site", () => { + it("source-grep — the failure notice is built through the locale seam, not a literal", () => { const stripped = readPostExecStripped(); - // The notice text must appear in non-comment source. - expect(stripped).toMatch(/\[tool failure\]/); + // It used to be a bare English `[tool failure] reported an error` + // appended raw to a reply in any language — a bracket-tagged internal + // string sitting outside the only mechanism that can translate it. + expect(stripped).toMatch(/buildToolFailureNotice\(/); + expect(stripped).not.toMatch(/\[tool failure\]/); + // The tool NAME still rides along verbatim (identifiers are never translated). + expect(stripped).toMatch(/failedToolName/); + }); + + it("source-grep — the notice never names the background poller as the culprit", () => { + const stripped = readPostExecStripped(); + // The poller relays OTHER tools' failures, so naming it points the reader at + // the one tool that was working — the same mis-attribution the retry + // breaker had. + expect(stripped).toMatch(/BACKGROUND_POLLER_TOOL/); }); it("source-grep — isSilentResponse guards the failure notice append (not the endReason override)", () => { @@ -858,7 +871,7 @@ describe("tool-failure endReason and notice", () => { // The notice call site must consult the recovery-aware helper, not raw failedTools. expect(stripped).toMatch(/unrecoveredFailedToolNames/); // The notice append must be guarded by a non-empty unrecovered set. - const noticeBlock = stripped.match(/unrecovered[A-Za-z]*\s*\.length\s*>\s*0[\s\S]{0,400}?\[tool failure\]/); + const noticeBlock = stripped.match(/unrecovered[A-Za-z]*\s*\.length\s*>\s*0[\s\S]{0,600}?buildToolFailureNotice/); expect(noticeBlock).not.toBeNull(); }); }); diff --git a/packages/agent/src/executor/executor-post-execution.ts b/packages/agent/src/executor/executor-post-execution.ts index c46d24994d..d3dafd939c 100644 --- a/packages/agent/src/executor/executor-post-execution.ts +++ b/packages/agent/src/executor/executor-post-execution.ts @@ -121,7 +121,8 @@ import { createHash, randomUUID } from "node:crypto"; // Critic hook (no inline logic — all logic in verification-gate.ts) import { shouldRunCritic, runVerificationCritic } from "./verification-gate.js"; // Deterministic user-facing replies for named degraded terminal causes. -import { buildOutputStarvedAnnotation, buildContextExhaustedReply, buildLoopDetectedReply, catalogFromLocalePacks, LOCALE_MESSAGE_IDS } from "./degraded-reply.js"; +import { buildOutputStarvedAnnotation, buildContextExhaustedReply, buildLoopDetectedReply, buildToolFailureNotice, catalogFromLocalePacks, LOCALE_MESSAGE_IDS } from "./degraded-reply.js"; +import { BACKGROUND_POLLER_TOOL } from "../safety/background-failure-attribution.js"; import { parseContextExhaustionCause } from "../context-engine/errors.js"; import { buildSyntheticCriticDeps } from "./verification-gate-synth-deps.js"; import { resolveScaffoldDefaults } from "./scaffold-defaults.js"; @@ -1370,6 +1371,22 @@ export async function postExecution(params: PostExecutionParams): Promise // acknowledged the failure or the response is a silent sentinel. The // observability label (effectiveFinishReason) is unchanged — operators still // see the recovered failure in logs/system. + const replyLanguage = params.responseLocalePolicy.locale; + // Wire the locale seam to operator config. `createLocaleCatalog` had exactly + // one production caller — the no-packs DEFAULT_LOCALE_CATALOG — so every + // deterministic reply resolved English no matter what `language` was pinned, + // and the documented "consumed by the deterministic degraded replies" claim + // had nothing behind it. An unknown id is reported, never silently kept. + const localeCatalog = catalogFromLocalePacks(config.localePacks, (locale, messageId) => { + deps.logger.warn( + { + step: "degraded-reply", + errorKind: "config" as const, + hint: `agents..localePacks.${locale}.${messageId} is not a platform-reply message id, so it is ignored; valid ids: ${LOCALE_MESSAGE_IDS.join(", ")}`, + }, + "unknown locale pack message id ignored", + ); + }); const unrecoveredFailed = unrecoveredFailedToolNames( bridgeResult.failedTools ?? [], bridgeResult.toolExecResults, @@ -1380,12 +1397,22 @@ export async function postExecution(params: PostExecutionParams): Promise !modelAcknowledgedFailure(result.response ?? "", unrecoveredFailed) && !isSilentResponse(result.response ?? "") ) { - const failedToolName = unrecoveredFailed[0]; + // Never NAME the background poller as the culprit. It relays other tools' + // failures, so blaming it points the reader at the one tool that was + // working — the same mis-attribution the retry breaker had. Prefer any + // real tool in the list; fall back to a nameless notice. + const failedToolName = unrecoveredFailed.find((t) => t !== BACKGROUND_POLLER_TOOL); + // Localized prose + the tool name VERBATIM. This was a bare English + // `[tool failure] reported an error`, appended raw to replies in + // any language — a bracket-tagged internal string, outside the only + // mechanism that can translate it. Identifiers stay untranslated by + // design; only the sentence around them is localized. // No "(see session log)" pointer: the recipient is the CHAT user, who has // no session log to see — the operator's lens is `comis explain` (the // failure rides the trajectory + IncidentReport.failures already). - result.response = (result.response ?? "") + - `\n[tool failure] ${failedToolName} reported an error`; + result.response = (result.response ?? "") + + buildToolFailureNotice(replyLanguage, localeCatalog) + + (failedToolName ?? ""); } // Degrade loudly — deliver an honest user-facing reply for named degraded causes. @@ -1394,22 +1421,6 @@ export async function postExecution(params: PostExecutionParams): Promise // Resolve the open response-locale policy once and pass the canonical tag to // each deterministic degraded-reply builder. Missing locale packs fall back // to the injected catalog's English strings. - const replyLanguage = params.responseLocalePolicy.locale; - // Wire the locale seam to operator config. `createLocaleCatalog` had exactly - // one production caller — the no-packs DEFAULT_LOCALE_CATALOG — so every - // deterministic reply resolved English no matter what `language` was pinned, - // and the documented "consumed by the deterministic degraded replies" claim - // had nothing behind it. An unknown id is reported, never silently kept. - const localeCatalog = catalogFromLocalePacks(config.localePacks, (locale, messageId) => { - deps.logger.warn( - { - step: "degraded-reply", - errorKind: "config" as const, - hint: `agents..localePacks.${locale}.${messageId} is not a platform-reply message id, so it is ignored; valid ids: ${LOCALE_MESSAGE_IDS.join(", ")}`, - }, - "unknown locale pack message id ignored", - ); - }); if (effectiveFinishReason === "output_starved") { result.response = (result.response ?? "") + buildOutputStarvedAnnotation(replyLanguage, localeCatalog); deps.logger.warn( From 6976804581a3ed29ff33004fa9f6a9e0d9e995de Mon Sep 17 00:00:00 2001 From: comis-agent Date: Mon, 27 Jul 2026 12:10:29 +0300 Subject: [PATCH 15/33] fix(agent): state the delegation trigger where the model can actually read it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The "## Task Delegation" policy — the >30s rule, the media/multi-file/deep- research criteria, the parallel fan-out instruction — reaches the model through exactly one path: SYSTEM_PROMPT_GUIDES["sessions_spawn"], which jit-guide-injector appends to a tool result only after sessions_spawn has been called successfully. That ordering is circular. The policy exists to make the model reach for the tool, and it is unreachable until the model has already reached for the tool. buildTaskDelegationSection() holds the same text with no production caller, so nothing puts any of it in the system prompt. The only always-present text was the lean description, "Start a background sub-agent and return its run ID immediately." — what the tool does, never when to use it. Observed live: an agent with sessions_spawn on its surface ran ten consecutive turns of heavy report work inline, worst turn 205s, and spawned zero sub-agents. Auto-backgrounding fired three times, which is the runtime rescuing work the model should have delegated. Move the trigger into the lean description, where it is present before the first call and costs one line instead of the ~2000-token guide. The full guide keeps its JIT delivery. --- docs/agent-tools/sessions.mdx | 2 ++ .../sections/tool-descriptions.test.ts | 31 +++++++++++++++++++ .../bootstrap/sections/tool-descriptions.ts | 10 +++++- 3 files changed, 42 insertions(+), 1 deletion(-) diff --git a/docs/agent-tools/sessions.mdx b/docs/agent-tools/sessions.mdx index 125b51a3f0..1159bb5c8c 100644 --- a/docs/agent-tools/sessions.mdx +++ b/docs/agent-tools/sessions.mdx @@ -160,6 +160,8 @@ The four session tools (`session_search`, `session_status`, `sessions_history`, | `max_steps` | integer | No | Maximum execution steps (floor of 30, default 50, capped at config default) | `sessions_spawn` never blocks for the child result. It returns the run ID and a background-running marker immediately; a queued spawn additionally returns `queued: true`. The result is delivered to the immutable originating conversation, so the model-facing tool cannot select an announcement channel. Use `subagents` with `action: wait` to await owned children without polling. + **When the agent reaches for it.** The tool description carries the delegation trigger, so an agent that has `sessions_spawn` on its surface is told up front to delegate rather than work inline whenever a task needs more than ~30 seconds of tool time, generates media, writes three or more files, requires deep research, or runs four or more dependent steps. Independent subtasks are spawned as multiple calls in one response and run in parallel. The fuller delegation guide (goal-oriented task phrasing, what not to delegate) is injected after the first successful spawn. + Sub-agent results are automatically condensed (if over the token threshold) and formatted with metadata tags before being returned to the parent. See [Subagent Context Lifecycle](/agents/subagent-lifecycle) for the full pipeline. **Example -- Spawn a coding agent:** diff --git a/packages/agent/src/bootstrap/sections/tool-descriptions.test.ts b/packages/agent/src/bootstrap/sections/tool-descriptions.test.ts index 440281e18b..3b498821c1 100644 --- a/packages/agent/src/bootstrap/sections/tool-descriptions.test.ts +++ b/packages/agent/src/bootstrap/sections/tool-descriptions.test.ts @@ -596,3 +596,34 @@ describe("providers_manage TOOL_GUIDE catalog interpolation", () => { ); }); }); + +// --------------------------------------------------------------------------- +// Deferred-section reachability +// --------------------------------------------------------------------------- + +/** + * A SYSTEM_PROMPT_GUIDES entry keyed on a tool name is injected by + * jit-guide-injector ONLY after that tool has been called successfully. When the + * guide's whole purpose is to tell the model WHEN to reach for the tool, that + * ordering is circular: the policy is unreachable until the model has already + * done the thing the policy exists to cause. + * + * The always-present lean description is the only text the model sees before + * its first call, so it must carry the trigger itself. Observed live: an agent + * with sessions_spawn on its surface ground a multi-minute report inline across + * ten consecutive turns and never spawned once, because nothing in the prompt + * said delegation applied. + */ +describe("SYSTEM_PROMPT_GUIDES trigger reachability", () => { + it("states the delegation trigger in the always-present sessions_spawn description", () => { + const lean = LEAN_TOOL_DESCRIPTIONS.sessions_spawn; + expect(typeof lean).toBe("string"); + // The >30s rule is the criterion that fires for heavy report work. + expect(lean as string).toMatch(/30\s*(?:s\b|sec)/i); + }); + + it("names parallel fan-out in the always-present sessions_spawn description", () => { + const lean = LEAN_TOOL_DESCRIPTIONS.sessions_spawn; + expect(lean as string).toMatch(/parallel|multiple/i); + }); +}); diff --git a/packages/agent/src/bootstrap/sections/tool-descriptions.ts b/packages/agent/src/bootstrap/sections/tool-descriptions.ts index 4ef7233d92..8228ed19e6 100644 --- a/packages/agent/src/bootstrap/sections/tool-descriptions.ts +++ b/packages/agent/src/bootstrap/sections/tool-descriptions.ts @@ -149,7 +149,15 @@ export const LEAN_TOOL_DESCRIPTIONS: Record30s of tool time, media generation, 3+ file writes," + + " deep research, or 4+ dependent steps; call it multiple times in one response for" + + " parallel subtasks.", subagents: "List, wait for, steer, or kill sub-agent runs for this session.", pipeline: "Define, execute, monitor, and cancel multi-node DAG execution graphs.", session_status: "Show agent status card: usage, model, steps. Optional per-session model override.", From f0e844d21f021194ebcf324a9338be4999cb862b Mon Sep 17 00:00:00 2001 From: comis-agent Date: Mon, 27 Jul 2026 12:47:43 +0300 Subject: [PATCH 16/33] fix(agent): stop redacting non-secret *_token arguments into the replay MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit SENSITIVE_ARG_NAMES matches the bare word `token` on any component boundary, so `range_token` — a fixed date-range enum on an MCP tool — was rewritten to "[REDACTED]" when the session was persisted. The matcher is name-based and value-blind, so nothing about the actual value could save it. That placeholder is not inert. It is replayed into the model's context on the next turn and copied forward as a literal argument, which is the exact hazard scrub-redacted-tool-calls.ts already documents for `env_value`. Live drive of ten prompts: every one of the five tool failures was this, across two different tools, with the pagination pattern making it unmistakable — pages 1, 2 and 3 rejected identically as the model re-sent "[REDACTED]". Split the matcher. Unambiguous credential names (api_key, secret, password, and qualified forms like access_token / refresh_token) stay value-blind, as does a parameter named exactly `token`. The ambiguous `*_token` family is redacted only when the value could plausibly be a credential; a short all-lowercase identifier cannot be one. looksLikeApiKey still runs afterwards as a second line of defence. Also name the accepted values on an enum rejection. AJV's "must be equal to one of the allowed values" lists nothing, so a model that sent a wrong value has nothing to correct toward and retries the same argument — which is why the same call failed three times rather than once. The values are already in the tool's schema at the format site, so the formatter now takes a resolver and interpolates them. Callers that pass no resolver keep the previous wording byte-identical. --- .../validation-error-formatter.ts | 31 ++++++++- .../safety/validation-error-formatter.test.ts | 51 +++++++++++++++ .../src/safety/validation-error-formatter.ts | 30 +++++++-- .../session/sanitize-session-secrets.test.ts | 64 ++++++++++++++++++- .../src/session/sanitize-session-secrets.ts | 54 ++++++++++++++-- 5 files changed, 220 insertions(+), 10 deletions(-) diff --git a/packages/agent/src/executor/stream-wrappers/validation-error-formatter.ts b/packages/agent/src/executor/stream-wrappers/validation-error-formatter.ts index 95db68a8ac..7aa7f0a886 100644 --- a/packages/agent/src/executor/stream-wrappers/validation-error-formatter.ts +++ b/packages/agent/src/executor/stream-wrappers/validation-error-formatter.ts @@ -31,6 +31,32 @@ import { formatValidationError } from "../../safety/validation-error-formatter.j * @param logger - Logger for debug output when a validation error is reformatted * @returns A named StreamFnWrapper ("validationErrorFormatter") */ +/** + * Build the enum-value resolver for one failing tool call by reading the tool's + * own parameter schema. Returns undefined when the tool or the parameter is not + * an enum, which keeps the formatter on its generic wording. + * + * Only top-level parameters are resolved — the nested case would need to walk + * the dot/bracket path, and every enum observed in practice sits at the top. + */ +function makeAllowedValuesResolver( + tools: unknown, + toolName: string | undefined, +): ((parameterPath: string) => string[] | undefined) | undefined { + if (!Array.isArray(tools) || toolName === undefined) return undefined; + const tool = (tools as Array<{ name?: string; parameters?: unknown }>) + .find((t) => t?.name === toolName); + const properties = (tool?.parameters as { properties?: Record } | undefined) + ?.properties; + if (properties === undefined) return undefined; + return (parameterPath) => { + const schema = properties[parameterPath] as { enum?: unknown } | undefined; + const values = schema?.enum; + if (!Array.isArray(values) || values.length === 0) return undefined; + return values.map((v) => String(v)); + }; +} + export function createValidationErrorFormatter( logger: ComisLogger, ): StreamFnWrapper { @@ -49,7 +75,10 @@ export function createValidationErrorFormatter( return msg; } - const formatted = formatValidationError(textBlock.text); + const formatted = formatValidationError( + textBlock.text, + makeAllowedValuesResolver(context.tools, msg.toolName), + ); if (formatted === null) { return msg; } diff --git a/packages/agent/src/safety/validation-error-formatter.test.ts b/packages/agent/src/safety/validation-error-formatter.test.ts index 0f3092638b..74d8feb442 100644 --- a/packages/agent/src/safety/validation-error-formatter.test.ts +++ b/packages/agent/src/safety/validation-error-formatter.test.ts @@ -327,3 +327,54 @@ describe("formatValidationError", () => { ); }); }); + +// --------------------------------------------------------------------------- +// Enum errors must name the values +// --------------------------------------------------------------------------- + +/** + * AJV's enum message ("must be equal to one of the allowed values") names no + * values, so a model that sent a wrong one has nothing to correct toward and + * simply retries the same argument. Observed live: three consecutive identical + * failures while paginating an MCP tool, because the rejected value was never + * contrasted with the accepted set. + * + * The allowed values are already in the tool's own schema at the call site, so + * the formatter takes a resolver and interpolates them. + */ +describe("formatValidationError — enum allowed values", () => { + const input = [ + 'Validation failed for tool "trips_search":', + " - range_token: must be equal to one of the allowed values", + "", + "Received arguments:", + '{ "range_token": "[REDACTED]" }', + ].join("\n"); + + it("lists the allowed values when the resolver supplies them", () => { + const out = formatValidationError(input, () => ["today", "yesterday", "last_week"]); + expect(out).toContain("today"); + expect(out).toContain("last_week"); + }); + + it("passes the failing parameter name to the resolver", () => { + const seen: string[] = []; + formatValidationError(input, (param) => { + seen.push(param); + return undefined; + }); + expect(seen).toContain("range_token"); + }); + + it("falls back to the generic wording when no values are available", () => { + expect(formatValidationError(input, () => undefined)).toBe( + "[trips_search] Invalid parameters:\n- `range_token` must be one of the allowed values", + ); + }); + + it("stays byte-identical with no resolver at all", () => { + expect(formatValidationError(input)).toBe( + "[trips_search] Invalid parameters:\n- `range_token` must be one of the allowed values", + ); + }); +}); diff --git a/packages/agent/src/safety/validation-error-formatter.ts b/packages/agent/src/safety/validation-error-formatter.ts index 8668dea79f..6e1f841a3e 100644 --- a/packages/agent/src/safety/validation-error-formatter.ts +++ b/packages/agent/src/safety/validation-error-formatter.ts @@ -17,6 +17,12 @@ // Header regex // --------------------------------------------------------------------------- +/** + * Resolves the accepted values for a failing parameter, given its dot-notation + * path. Returns undefined when the parameter is not a known enum. + */ +export type AllowedValuesResolver = (parameterPath: string) => string[] | undefined; + /** Matches the pi-ai validation error header line. */ const HEADER_RE = /^Validation failed for tool "([^"]+)":/; @@ -74,7 +80,11 @@ function convertInstancePath(path: string): string { /** * Rewrite a single AJV error (path + message) into LLM-friendly text. */ -function rewriteErrorMessage(path: string, message: string): string { +function rewriteErrorMessage( + path: string, + message: string, + allowedValuesFor?: AllowedValuesResolver, +): string { const displayPath = convertInstancePath(path); // "must have required property 'X'" -> "Required parameter `X` is missing" @@ -96,8 +106,14 @@ function rewriteErrorMessage(path: string, message: string): string { return `\`${displayPath}\` expected ${typeMatch[1]}`; } - // "must be equal to one of the allowed values" -> simplified + // "must be equal to one of the allowed values" -> name the values when the + // caller can resolve them from the tool's own schema. AJV's wording omits + // them, which leaves a model retrying the same rejected argument. if (ENUM_RE.test(message)) { + const allowed = allowedValuesFor?.(displayPath); + if (allowed !== undefined && allowed.length > 0) { + return `\`${displayPath}\` must be one of: ${allowed.join(", ")}`; + } return `\`${displayPath}\` must be one of the allowed values`; } @@ -119,10 +135,16 @@ function rewriteErrorMessage(path: string, message: string): string { * concise, LLM-friendly format. * * @param errorText - The raw error string from a tool result + * @param allowedValuesFor - Optional resolver returning the accepted values for + * a failing parameter, so an enum rejection can name them. Omitting it + * keeps the previous generic wording byte-identical. * @returns Reformatted error string, or `null` if the text is not a * validation error matching the pi-ai pattern */ -export function formatValidationError(errorText: string): string | null { +export function formatValidationError( + errorText: string, + allowedValuesFor?: AllowedValuesResolver, +): string | null { // Quick exit for non-validation errors const headerMatch = HEADER_RE.exec(errorText); if (!headerMatch) return null; @@ -145,7 +167,7 @@ export function formatValidationError(errorText: string): string | null { const path = match[1]!; const message = match[2]!; - rewritten.push(rewriteErrorMessage(path, message)); + rewritten.push(rewriteErrorMessage(path, message, allowedValuesFor)); } if (rewritten.length === 0) return null; diff --git a/packages/agent/src/session/sanitize-session-secrets.test.ts b/packages/agent/src/session/sanitize-session-secrets.test.ts index e035496d93..d64c829f93 100644 --- a/packages/agent/src/session/sanitize-session-secrets.test.ts +++ b/packages/agent/src/session/sanitize-session-secrets.test.ts @@ -1,6 +1,10 @@ // SPDX-License-Identifier: Apache-2.0 import { describe, it, expect, beforeEach, afterEach } from "vitest"; -import { sanitizeSessionSecrets, looksLikeApiKey } from "./sanitize-session-secrets.js"; +import { + sanitizeSessionSecrets, + looksLikeApiKey, + projectSessionValueForPersistence, +} from "./sanitize-session-secrets.js"; import { mkdtempSync, writeFileSync, readFileSync, rmSync, statSync } from "node:fs"; import { join } from "node:path"; import { tmpdir } from "node:os"; @@ -546,3 +550,61 @@ describe("looksLikeApiKey", () => { expect(looksLikeApiKey("[REDACTED]")).toBe(false); }); }); + +// --------------------------------------------------------------------------- +// Bare-`token` name collision +// --------------------------------------------------------------------------- + +/** + * The sensitive-argument matcher is name-based and value-blind, so the bare + * `token` keyword captures any parameter ending in `_token` — including opaque + * cursors and plain enums that never carry a credential. + * + * Redacting one of those is not merely over-cautious, it is corrupting: the + * placeholder is persisted into the session, replayed back into the model's + * context on the next turn, and copied forward as a literal argument. That is + * the same replay hazard scrub-redacted-tool-calls.ts already documents for + * `env_value`. Observed live: an MCP tool whose `range_token` parameter is a + * fixed date-range enum failed schema validation three times in a row as the + * model paginated, each call re-sending "[REDACTED]". + * + * Values that could plausibly be a credential must still be redacted on these + * names — the exemption is for values that cannot be one. + */ +describe("projectSessionValueForPersistence — bare-token name collision", () => { + it("keeps a short lowercase enum on a *_token parameter", () => { + const out = projectSessionValueForPersistence({ range_token: "last_7_days" }); + expect(out.value).toEqual({ range_token: "last_7_days" }); + expect(out.redactions).toBe(0); + }); + + it("keeps an opaque pagination cursor that cannot be a credential", () => { + const out = projectSessionValueForPersistence({ page_token: "next_page" }); + expect(out.value).toEqual({ page_token: "next_page" }); + }); + + it("still redacts a credential-shaped value on a *_token parameter", () => { + const out = projectSessionValueForPersistence({ + range_token: "sk-AbC123XyZ456DeF789GhI012JkL345MnO", + }); + expect((out.value as Record).range_token).toBe("[REDACTED]"); + expect(out.redactions).toBe(1); + }); + + it("still redacts qualified credential token names regardless of value shape", () => { + for (const name of ["access_token", "auth_token", "refresh_token", "id_token"]) { + const out = projectSessionValueForPersistence({ [name]: "last_7_days" }); + expect( + (out.value as Record)[name], + `${name} must stay redacted`, + ).toBe("[REDACTED]"); + } + }); + + it("still redacts the unqualified api_key/secret/password family by name", () => { + for (const name of ["api_key", "secret", "password", "private_key"]) { + const out = projectSessionValueForPersistence({ [name]: "short_value" }); + expect((out.value as Record)[name]).toBe("[REDACTED]"); + } + }); +}); diff --git a/packages/agent/src/session/sanitize-session-secrets.ts b/packages/agent/src/session/sanitize-session-secrets.ts index c0edc491f3..6de19d91c9 100644 --- a/packages/agent/src/session/sanitize-session-secrets.ts +++ b/packages/agent/src/session/sanitize-session-secrets.ts @@ -43,7 +43,52 @@ const API_KEY_PATTERNS: RegExp[] = [ /** Argument names that are always sensitive regardless of value pattern. */ const SENSITIVE_ARG_NAMES = - /(?:^|[_-])(?:api[_-]?key|apikey|token|secret|password|credential|auth[_-]?key|access[_-]?key|private[_-]?key|username|env[_-]?value)(?:$|[_-])/i; + /(?:^|[_-])(?:api[_-]?key|apikey|secret|password|credential|auth[_-]?key|access[_-]?key|private[_-]?key|username|env[_-]?value|(?:access|auth|refresh|bearer|id|api|session|oauth|csrf|xsrf)[_-]?token)(?:$|[_-])/i; + +/** + * A parameter named exactly `token` — no qualifier to disambiguate it. Treated + * as always-sensitive: on its own the word most often names a credential. + */ +const BARE_TOKEN_ARG_NAME = /^token$/i; + +/** + * Names carrying `token` as a component without a credential qualifier — + * `range_token`, `page_token`, `continuation_token`. These are usually opaque + * cursors or plain enums, so they are redacted only when the VALUE could + * plausibly be a credential. + * + * Redacting one of these unconditionally is corrupting rather than merely + * cautious: the placeholder is persisted, replayed into the model's context on + * the next turn, and copied forward as a literal argument — the replay hazard + * scrub-redacted-tool-calls.ts documents for `env_value`. Observed live against + * an MCP tool whose `range_token` is a fixed date-range enum: three consecutive + * schema-validation failures while paginating, each re-sending "[REDACTED]". + */ +const QUALIFIED_TOKEN_ARG_NAMES = /(?:^|[_-])token(?:$|[_-])/i; + +/** + * True when a value cannot plausibly be a credential: a short, all-lowercase + * identifier such as an enum member or a cursor keyword. Anything longer, + * mixed-case, or shaped like an encoded digest fails this test and stays + * redacted. `looksLikeApiKey` still runs afterwards as a second line of defence. + */ +function isImplausibleSecretValue(value: string): boolean { + if (value.length === 0 || value.length > 24) return false; + if (!/^[a-z][a-z0-9]*(?:[_-][a-z0-9]+)*$/.test(value)) return false; + if (/^[a-f0-9]{16,}$/.test(value)) return false; // lowercase hex digest + return true; +} + +/** + * Whether an argument must be redacted on the strength of its NAME. + * Value-blind for unambiguous credential names; value-gated for the ambiguous + * `*_token` family so a public enum is not destroyed. + */ +function isSensitiveArgName(name: string, value: string): boolean { + if (SENSITIVE_ARG_NAMES.test(name) || BARE_TOKEN_ARG_NAME.test(name)) return true; + if (QUALIFIED_TOKEN_ARG_NAMES.test(name)) return !isImplausibleSecretValue(value); + return false; +} // eslint-disable-next-line no-restricted-syntax -- durable-session redaction sentinel (not the Pino censor literal) const REDACTION_PLACEHOLDER = "[REDACTED]"; @@ -108,8 +153,8 @@ function projectPersistenceValue( if (typeof value === "string") { if ( fieldName !== undefined - && SENSITIVE_ARG_NAMES.test(fieldName) && value.length > 0 + && isSensitiveArgName(fieldName, value) && !isSafePersistencePlaceholder(value) ) { return { value: REDACTION_PLACEHOLDER, redactions: 1 }; @@ -224,10 +269,11 @@ const SANITIZATION_RULES: SanitizationRule[] = [ match(_toolName, args) { let changed = false; for (const key of Object.keys(args)) { - if (SENSITIVE_ARG_NAMES.test(key)) { + { const val = args[key]; // eslint-disable-next-line no-restricted-syntax -- session sensitive-arg already-redacted sentinel (not the Pino censor literal) - if (typeof val === "string" && val !== "[REDACTED]" && val.length > 0) { + if (typeof val === "string" && val !== "[REDACTED]" && val.length > 0 + && isSensitiveArgName(key, val)) { // eslint-disable-next-line no-restricted-syntax -- session sensitive-arg redaction (not the Pino censor literal) args[key] = "[REDACTED]"; changed = true; From 93cf31d091d4a7ea46079dabc8f12838f0127d42 Mon Sep 17 00:00:00 2001 From: comis-agent Date: Mon, 27 Jul 2026 13:15:26 +0300 Subject: [PATCH 17/33] feat(observability): fingerprint the tool array in the cache trace An Anthropic cached prefix is system -> tools -> messages. The trace fingerprinted the first and third components and not the second, so a tool added, removed, or re-described mid-session invalidated every cached block after the system prompt while the record built to diagnose prefix collapse showed nothing moving. That gap blocked a live investigation: 37 calls on one drive held a single stable systemDigest and still burned 10.8M cache-creation tokens against 4.0M reads, creating a full ~350-450k prefix on nearly every call and reading it back eight times. System stable plus messages growing normally leaves the tool array as the remaining suspect, and the trace could not speak to it either way. Digest the full tool definitions rather than the name list, so a schema or description edit that leaves names intact still moves the digest. toolCount rides along for the cheap case where the array simply grew. --- .../src/cache-trace/stream-fn-wrapper.test.ts | 72 +++++++++++++++++++ .../src/cache-trace/stream-fn-wrapper.ts | 13 ++++ .../observability/src/cache-trace/types.ts | 4 ++ 3 files changed, 89 insertions(+) diff --git a/packages/observability/src/cache-trace/stream-fn-wrapper.test.ts b/packages/observability/src/cache-trace/stream-fn-wrapper.test.ts index d81109941a..5daef3b17a 100644 --- a/packages/observability/src/cache-trace/stream-fn-wrapper.test.ts +++ b/packages/observability/src/cache-trace/stream-fn-wrapper.test.ts @@ -382,3 +382,75 @@ describe("buildCacheTraceWrapper emits model:before stage", () => { expect(modelBefore!.systemDigest).toBe(streamContext!.systemDigest); }); }); + +// --------------------------------------------------------------------------- +// toolsDigest +// --------------------------------------------------------------------------- + +/** + * An Anthropic cached prefix is system -> tools -> messages, so a change to the + * tool array invalidates everything after the system block just as surely as a + * changed system prompt does. The trace fingerprinted system and messages but + * not tools, leaving one of the three prefix components invisible to the very + * record built to diagnose prefix collapse. + * + * Observed live: 37 calls with a single stable systemDigest, 10.8M cache-creation + * tokens against 4.0M reads, and no way to tell from the trace whether the tool + * array was moving underneath. + */ +describe("buildCacheTraceWrapper toolsDigest", () => { + function contextWithTools(tools: unknown[]): unknown { + return { + messages: [{ role: "user", content: "hello" }], + systemPrompt: "you are an assistant", + tools, + }; + } + + let digestSeq = 0; + async function digestFor(tools: unknown[]): Promise> { + // Unique per call — two variants can serialize to the same length, and a + // shared path would make both runs append to one file so `find` returns + // the first record for both. + const filePath = join(tmpDir, `tools-${digestSeq++}.jsonl`); + const trace = makeTrace({ includeMessages: false, filePath }); + const wrap = buildCacheTraceWrapper(trace); + const next: StreamFn = ((..._args: unknown[]) => + ({ usage: { cacheRead: 0, cacheWrite: 0 } }) as unknown as ReturnType) as StreamFn; + wrap(next)( + fakeModel() as Parameters[0], + contextWithTools(tools) as Parameters[1], + ); + await trace.flush(); + return readLines(filePath).find((l) => l.stage === "stream:context")!; + } + + it("emits a toolsDigest and a toolCount on stream:context", async () => { + const rec = await digestFor([{ name: "read" }, { name: "write" }]); + expect(rec.toolsDigest).toMatch(/^[0-9a-f]{64}$/); + expect(rec.toolCount).toBe(2); + }); + + it("keeps the digest stable for an unchanged tool array", async () => { + const a = await digestFor([{ name: "read" }, { name: "write" }]); + const b = await digestFor([{ name: "read" }, { name: "write" }]); + expect(a.toolsDigest).toBe(b.toolsDigest); + }); + + it("changes the digest when a tool is added", async () => { + const a = await digestFor([{ name: "read" }]); + const b = await digestFor([{ name: "read" }, { name: "grep" }]); + expect(a.toolsDigest).not.toBe(b.toolsDigest); + }); + + it("changes the digest when only a tool schema changes", async () => { + const a = await digestFor([{ name: "read", parameters: { a: 1 } }]); + const b = await digestFor([{ name: "read", parameters: { a: 2 } }]); + expect(a.toolsDigest).not.toBe(b.toolsDigest); + }); + + it("reports a zero toolCount when the context carries no tools", async () => { + const rec = await digestFor([]); + expect(rec.toolCount).toBe(0); + }); +}); diff --git a/packages/observability/src/cache-trace/stream-fn-wrapper.ts b/packages/observability/src/cache-trace/stream-fn-wrapper.ts index d4b0ba37d4..231593af12 100644 --- a/packages/observability/src/cache-trace/stream-fn-wrapper.ts +++ b/packages/observability/src/cache-trace/stream-fn-wrapper.ts @@ -237,12 +237,25 @@ export function buildCacheTraceWrapper(trace: CacheTrace): StreamFnWrapper { return typeof role === "string" ? role : "unknown"; }); + // The tool array is the SECOND component of an Anthropic cached prefix + // (system -> tools -> messages), so a tool added, removed, or re-described + // mid-session invalidates every cached block after the system prompt — + // exactly like a changed system prompt, and just as invisible without a + // fingerprint. Digesting the full definition (not just names) is what + // catches a schema or description edit that leaves the name list intact. + const tools = Array.isArray((context as { tools?: unknown }).tools) + ? ((context as { tools?: unknown[] }).tools as unknown[]) + : []; + const toolsDigest = sha256(tools.map((t) => sha256(stableStringify(t))).join("|")); + const preCallPayload: Record = { messageCount: messages.length, messageRoles, messageFingerprints, messagesDigest, systemDigest, + toolCount: tools.length, + toolsDigest, }; // The SMALL assembled-array shape descriptor (counts/flags + tool // id pairing). Emitted OUTSIDE the includeMessages guard so it is diff --git a/packages/observability/src/cache-trace/types.ts b/packages/observability/src/cache-trace/types.ts index 27ae3ef594..329385966c 100644 --- a/packages/observability/src/cache-trace/types.ts +++ b/packages/observability/src/cache-trace/types.ts @@ -152,6 +152,10 @@ export const CacheTraceEventSchema = z.object({ messagesDigest: z.string().optional(), system: z.unknown().optional(), systemDigest: z.string().optional(), + // The tool array — the prefix component between system and messages. A + // change here invalidates every cached block after the system prompt. + toolCount: z.number().int().nonnegative().optional(), + toolsDigest: z.string().optional(), // Token attribution. `session:after` aggregates across the session via // the EventBus bridge stash; `model:after` carries the per-call snapshot // from the StreamFn return value's usage block. From d77556fa7762ab6bcf1a7a1ba31580b905a781ee Mon Sep 17 00:00:00 2001 From: comis-agent Date: Mon, 27 Jul 2026 13:29:54 +0300 Subject: [PATCH 18/33] fix(agent): stop asking Bedrock for a cache TTL it cannot honor MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The 1-hour cache TTL is an Anthropic-API beta. Bedrock serves Anthropic models but does not accept Anthropic beta headers — request-body/factory.ts already gates the 1M-context beta on the direct provider for exactly this reason, with the comment "direct Anthropic only -- NOT Bedrock/Vertex". The body-level `ttl: "1h"` carried no equivalent gate, so every cached block went out asking for an hour on a provider that cannot grant it. The write is billed; the read never matches. Measured on amazon-bedrock/claude-opus-4-8 with the new toolsDigest in place: a single stable systemDigest, a single stable toolsDigest over a constant 193-tool array, ~459k cache-creation tokens on every call, and zero cache reads across the entire drive. Nothing in the prefix was moving — the same two prefixes were re-created call after call and never once matched. Cap retention at the one point where it is resolved, so all four downstream sites that write `ttl: "1h"` follow from the same decision rather than each growing its own provider guard. supportsExtendedCacheTtl is deliberately NOT the family check: Bedrock IS in the Anthropic family, which is why the family predicate cannot express this. getCacheProviderInfo has the same literal-string gap but no production consumers, and an existing test deliberately pins Bedrock to catalog-driven eligibility. Left alone rather than fought for no runtime effect. --- .../breakpoint-orchestration.test.ts | 52 +++++++++++++++++++ .../request-body/breakpoint-orchestration.ts | 27 +++++++++- .../agent/src/provider/capabilities.test.ts | 38 ++++++++++++++ packages/agent/src/provider/capabilities.ts | 18 +++++++ 4 files changed, 134 insertions(+), 1 deletion(-) diff --git a/packages/agent/src/executor/stream-wrappers/request-body/breakpoint-orchestration.test.ts b/packages/agent/src/executor/stream-wrappers/request-body/breakpoint-orchestration.test.ts index aa3cdda296..83fa14b4b5 100644 --- a/packages/agent/src/executor/stream-wrappers/request-body/breakpoint-orchestration.test.ts +++ b/packages/agent/src/executor/stream-wrappers/request-body/breakpoint-orchestration.test.ts @@ -268,3 +268,55 @@ describe("runCacheBreakpointPhase — UNTRUSTED_ block 1h cache anchor", () => { expect(violations).toEqual([]); }); }); + +// --------------------------------------------------------------------------- +// Provider retention cap +// --------------------------------------------------------------------------- + +/** + * Bedrock and Vertex serve Anthropic models but cannot accept the Anthropic beta + * that enables the 1-hour cache TTL. A `ttl: "1h"` marker sent there is billed as + * a cache write and never matched on the following request. + * + * Measured live on amazon-bedrock with a byte-stable system prompt AND a + * byte-stable 193-tool array: ~459k cache-creation tokens on every call and zero + * cache reads for the entire drive. + * + * The cap belongs at this single resolution point — the four downstream sites + * that write `ttl: "1h"` all read the retention decided here. + */ +describe("runCacheBreakpointPhase — extended-TTL provider cap", () => { + function retentionFor(provider: string) { + return runCacheBreakpointPhase( + makeResult([{ role: "user", content: [{ type: "text", text: "hi" }] }]), + { id: "claude-opus-4-8", provider }, + makeConfig({ getCacheRetention: () => "long" }), + true, + false, + 1024, + makeLogger(), + ); + } + + it("keeps long retention on direct Anthropic", () => { + expect(retentionFor("anthropic")).toBe("long"); + }); + + it("caps long retention to short on amazon-bedrock", () => { + expect(retentionFor("amazon-bedrock")).toBe("short"); + }); + + it("emits no 1h marker on the system block for bedrock", () => { + const result = makeResult([{ role: "user", content: [{ type: "text", text: "hi" }] }]); + runCacheBreakpointPhase( + result, + { id: "claude-opus-4-8", provider: "amazon-bedrock" }, + makeConfig({ getCacheRetention: () => "long" }), + true, + false, + 1024, + makeLogger(), + ); + expect(JSON.stringify(result)).not.toContain('"1h"'); + }); +}); diff --git a/packages/agent/src/executor/stream-wrappers/request-body/breakpoint-orchestration.ts b/packages/agent/src/executor/stream-wrappers/request-body/breakpoint-orchestration.ts index e662eb89a5..2e5c9e1830 100644 --- a/packages/agent/src/executor/stream-wrappers/request-body/breakpoint-orchestration.ts +++ b/packages/agent/src/executor/stream-wrappers/request-body/breakpoint-orchestration.ts @@ -27,6 +27,7 @@ import type { ComisLogger } from "@comis/core"; import { SYSTEM_PROMPT_DYNAMIC_BOUNDARY, resolveBreakpointStrategy } from "../config-resolver.js"; import { djb2 } from "../../cache-detection/index.js"; +import { supportsExtendedCacheTtl } from "../../../provider/capabilities.js"; import { addCacheControlToLastBlock } from "./cache-control-block.js"; import { placeCacheBreakpoints } from "./breakpoint-placement.js"; @@ -65,8 +66,32 @@ export function runCacheBreakpointPhase( config.getCacheRetentionOverrides?.(), ); + // Downgrade "long" to "short" on a provider that cannot honor the extended + // TTL beta (Bedrock/Vertex). Done HERE, at the single point where retention is + // resolved, so every downstream `=== "long"` site emits a plain 5m marker — + // rather than adding a parallel provider guard at each of the four places that + // write `ttl: "1h"`. An unhonored 1h marker is billed as a cache write and + // never matched on the next request. + const providerCappedRetention: CacheRetention = + effectiveRetention === "long" && !supportsExtendedCacheTtl(model.provider) + ? "short" + : effectiveRetention; + if (providerCappedRetention !== effectiveRetention) { + logger.debug( + { + provider: model.provider, + modelId, + requested: effectiveRetention, + applied: providerCappedRetention, + hint: "provider does not support the extended cache TTL beta; using the 5m default", + step: "cache-retention", + }, + "Cache retention capped for provider", + ); + } + // Latch retention on first resolution - const rawRetention = effectiveRetention; + const rawRetention = providerCappedRetention; const retentionLatch = config.getRetentionLatch?.(); const resolvedRetention: CacheRetention = retentionLatch ? retentionLatch.setOnce(rawRetention) diff --git a/packages/agent/src/provider/capabilities.test.ts b/packages/agent/src/provider/capabilities.test.ts index 896eb05e07..13abcf8640 100644 --- a/packages/agent/src/provider/capabilities.test.ts +++ b/packages/agent/src/provider/capabilities.test.ts @@ -23,6 +23,7 @@ import { resolveProviderCapabilities, normalizeProviderId, isAnthropicFamily, + supportsExtendedCacheTtl, isOpenAiFamily, isGoogleFamily, isGoogleAIStudio, @@ -458,3 +459,40 @@ describe("validateProviderOverrides", () => { expect(() => validateProviderOverrides({ warn })).not.toThrow(); }); }); + +// --------------------------------------------------------------------------- +// Extended (1-hour) cache TTL +// --------------------------------------------------------------------------- + +/** + * Anthropic's 1-hour cache TTL is an API beta. Bedrock and Vertex serve + * Anthropic models but do not accept Anthropic beta headers — the 1M-context + * beta is already gated to direct Anthropic for exactly this reason — so a + * `ttl: "1h"` marker sent to them cannot be honored. + * + * Observed live on amazon-bedrock: a byte-stable system prompt AND a byte-stable + * 193-tool array, ~459k cache-creation tokens on every single call, and zero + * cache reads across the whole drive. + */ +describe("supportsExtendedCacheTtl", () => { + it("returns true for direct Anthropic", () => { + expect(supportsExtendedCacheTtl("anthropic")).toBe(true); + }); + + it("returns false for Anthropic models served through Bedrock", () => { + expect(supportsExtendedCacheTtl("amazon-bedrock")).toBe(false); + expect(supportsExtendedCacheTtl("bedrock")).toBe(false); + }); + + it("returns false for non-Anthropic providers", () => { + expect(supportsExtendedCacheTtl("openai")).toBe(false); + expect(supportsExtendedCacheTtl("groq")).toBe(false); + }); + + it("agrees with isAnthropicFamily only for the direct provider", () => { + // Bedrock IS in the Anthropic family — that is precisely why the TTL gate + // cannot reuse the family check. + expect(isAnthropicFamily("amazon-bedrock")).toBe(true); + expect(supportsExtendedCacheTtl("amazon-bedrock")).toBe(false); + }); +}); diff --git a/packages/agent/src/provider/capabilities.ts b/packages/agent/src/provider/capabilities.ts index 10943f3dc5..81d5a02280 100644 --- a/packages/agent/src/provider/capabilities.ts +++ b/packages/agent/src/provider/capabilities.ts @@ -116,6 +116,24 @@ export function isAnthropicFamily(provider: string): boolean { return resolveProviderCapabilities(provider).providerFamily === "anthropic"; } +/** + * Whether a provider can honor Anthropic's extended (1-hour) cache TTL. + * + * Deliberately NOT the family check. Extended TTL is an Anthropic-API beta, and + * Bedrock/Vertex serve Anthropic models without accepting Anthropic beta headers + * — the 1M-context beta is gated to the direct provider for the same reason + * (request-body/factory.ts: "direct Anthropic only -- NOT Bedrock/Vertex"). + * + * Marking a block `ttl: "1h"` where the beta is unavailable produces a cache + * entry the provider will not match on the next request: the write is billed, + * the read never lands. Measured on amazon-bedrock with a byte-stable system + * prompt and a byte-stable 193-tool array — ~459k creation tokens per call, + * zero reads across the entire drive. + */ +export function supportsExtendedCacheTtl(provider: string): boolean { + return normalizeProviderId(provider) === "anthropic"; +} + /** * Check if a provider belongs to the OpenAI family. * True for: openai, azure-openai-responses, openai-codex (and their aliases). From d2bc4cf671812b10543e90e116a64b834b20602f Mon Sep 17 00:00:00 2001 From: comis-agent Date: Mon, 27 Jul 2026 14:06:06 +0300 Subject: [PATCH 19/33] fix(agent): stop the monotonic sweep from undoing the provider retention cap MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Capping retention at the resolution point was not enough. Three sites emit ttl: "1h" directly without reading the resolved retention — the UNTRUSTED_ anchor, adaptive zone promotion, and the safety-net sweep itself. The sweep runs last, sees one of those 1h markers, and upgrades every earlier 5m marker to match, silently reversing the cap on a provider that cannot honor it. Live on amazon-bedrock after the cap landed, this fired on EVERY turn, logging "upgraded out-of-order 5m markers to 1h" with errorKind:"internal" and a hint pointing at an upstream placement bug — while itself being the thing re-breaking the request. Make the sweep provider-aware. Where extended TTL is unavailable it now normalizes DOWNWARD, stripping every ttl so all markers are the plain 5m default: monotonicity holds trivially, every marker is honorable, and any future site that emits 1h directly is corrected rather than propagated. Demoted to DEBUG for that path — it is the expected steady state there, not a defect worth a WARN. Upgrade behaviour is unchanged and remains the default when the flag is omitted, so direct Anthropic keeps its 1h coordination byte-for-byte. --- .../request-body/breakpoint-orchestration.ts | 2 +- .../stream-wrappers/request-body/factory.ts | 4 +- .../request-body/monotonic-ttl.test.ts | 56 +++++++++++++++++++ .../request-body/monotonic-ttl.ts | 31 ++++++++++ 4 files changed, 90 insertions(+), 3 deletions(-) diff --git a/packages/agent/src/executor/stream-wrappers/request-body/breakpoint-orchestration.ts b/packages/agent/src/executor/stream-wrappers/request-body/breakpoint-orchestration.ts index 2e5c9e1830..673cc37d47 100644 --- a/packages/agent/src/executor/stream-wrappers/request-body/breakpoint-orchestration.ts +++ b/packages/agent/src/executor/stream-wrappers/request-body/breakpoint-orchestration.ts @@ -430,7 +430,7 @@ export function runCacheBreakpointPhase( // anchor-aware retention above correctly coordinates the source placement. // Logs WARN with errorKind:"internal" if any upgrade fires — that // indicates an upstream placement bug. - enforceMonotonicTtlOrdering(result, logger); + enforceMonotonicTtlOrdering(result, logger, supportsExtendedCacheTtl(model.provider)); return resolvedRetention; } diff --git a/packages/agent/src/executor/stream-wrappers/request-body/factory.ts b/packages/agent/src/executor/stream-wrappers/request-body/factory.ts index 516e7bae0c..9943580980 100644 --- a/packages/agent/src/executor/stream-wrappers/request-body/factory.ts +++ b/packages/agent/src/executor/stream-wrappers/request-body/factory.ts @@ -31,7 +31,7 @@ import type { ComisLogger } from "@comis/core"; import type { StreamFnWrapper } from "../types.js"; import { createAccumulativeLatch } from "../../session-latch.js"; -import { isAnthropicFamily } from "../../../provider/capabilities.js"; +import { isAnthropicFamily, supportsExtendedCacheTtl } from "../../../provider/capabilities.js"; import type { RequestBodyInjectorConfig } from "./types.js"; import { getMinCacheableTokens } from "./cache-breakpoints.js"; @@ -373,7 +373,7 @@ export function createRequestBodyInjector( // net so any stray 5m-before-1h gets upgraded before the // request leaves the wrapper. The sweep is a no-op when the // promoted layout happens to remain monotonic. - enforceMonotonicTtlOrdering(result, logger); + enforceMonotonicTtlOrdering(result, logger, supportsExtendedCacheTtl(model.provider)); } } } diff --git a/packages/agent/src/executor/stream-wrappers/request-body/monotonic-ttl.test.ts b/packages/agent/src/executor/stream-wrappers/request-body/monotonic-ttl.test.ts index e63f02702d..1547ec81db 100644 --- a/packages/agent/src/executor/stream-wrappers/request-body/monotonic-ttl.test.ts +++ b/packages/agent/src/executor/stream-wrappers/request-body/monotonic-ttl.test.ts @@ -295,3 +295,59 @@ describe("enforceMonotonicTtlOrdering", () => { expect(msg0Content[1]!.cache_control).toEqual({ type: "custom" }); }); }); + +// --------------------------------------------------------------------------- +// Provider without extended TTL +// --------------------------------------------------------------------------- + +/** + * On a provider that cannot honor the 1-hour beta (Bedrock), the sweep must + * normalize DOWNWARD: any 1h marker becomes 5m, which satisfies monotonicity + * trivially and keeps every marker honorable. + * + * Upgrading instead is actively harmful there — an unhonored 1h marker is billed + * as a cache write and never matched. Observed live after the retention cap + * landed: this sweep fired on EVERY turn ("upgraded out-of-order 5m markers to + * 1h"), silently undoing the cap, because sites like the UNTRUSTED_ anchor and + * adaptive zone promotion still emit 1h directly. + */ +describe("enforceMonotonicTtlOrdering — provider without extended TTL", () => { + function payload() { + return { + system: [{ type: "text", text: "s", cache_control: { type: "ephemeral" } }], + messages: [ + { + role: "user", + content: [{ type: "text", text: "m", cache_control: { type: "ephemeral", ttl: "1h" } }], + }, + ], + } as Record; + } + + it("downgrades 1h markers to 5m when extended TTL is unavailable", () => { + const result = payload(); + enforceMonotonicTtlOrdering(result, makeLogger(), false); + expect(JSON.stringify(result)).not.toContain("1h"); + }); + + it("leaves every marker ephemeral after the downgrade", () => { + const result = payload(); + enforceMonotonicTtlOrdering(result, makeLogger(), false); + const msgs = result.messages as Array<{ content: Array> }>; + expect(msgs[0]!.content[0]!.cache_control).toEqual({ type: "ephemeral" }); + }); + + it("still upgrades to 1h when extended TTL IS available", () => { + const result = payload(); + enforceMonotonicTtlOrdering(result, makeLogger(), true); + const sys = result.system as Array>; + expect(sys[0]!.cache_control).toEqual({ type: "ephemeral", ttl: "1h" }); + }); + + it("defaults to the upgrade behaviour when the flag is omitted", () => { + const result = payload(); + enforceMonotonicTtlOrdering(result, makeLogger()); + const sys = result.system as Array>; + expect(sys[0]!.cache_control).toEqual({ type: "ephemeral", ttl: "1h" }); + }); +}); diff --git a/packages/agent/src/executor/stream-wrappers/request-body/monotonic-ttl.ts b/packages/agent/src/executor/stream-wrappers/request-body/monotonic-ttl.ts index f0710c600b..730f501f36 100644 --- a/packages/agent/src/executor/stream-wrappers/request-body/monotonic-ttl.ts +++ b/packages/agent/src/executor/stream-wrappers/request-body/monotonic-ttl.ts @@ -124,8 +124,39 @@ function collectMarkers(result: Record): MarkerRef[] { export function enforceMonotonicTtlOrdering( result: Record, logger: ComisLogger, + allowExtendedTtl = true, ): void { const markers = collectMarkers(result); + if (markers.length === 0) return; + + // On a provider that cannot honor the 1h beta (Bedrock/Vertex), normalize + // DOWNWARD instead: strip every ttl so all markers are the plain 5m default. + // That satisfies monotonicity trivially AND keeps every marker honorable — + // upgrading here would silently undo the provider retention cap, because + // sites that emit 1h directly (the UNTRUSTED_ anchor, adaptive zone + // promotion) do not read the resolved retention. No WARN: this is the + // expected steady state for those providers, not an upstream placement bug. + if (!allowExtendedTtl) { + let downgraded = 0; + for (const m of markers) { + if (m.ttl !== "1h") continue; + m.block.cache_control = { type: "ephemeral" }; + downgraded++; + } + if (downgraded > 0) { + logger.debug( + { + downgradedCount: downgraded, + totalMarkers: markers.length, + hint: "provider does not support the extended cache TTL beta; markers normalized to the 5m default", + step: "cache-retention", + }, + "MONOTONIC-TTL: normalized 1h markers to 5m for provider", + ); + } + return; + } + if (markers.length <= 1) return; // Walk backward; once we see a 1h marker, every earlier 5m marker must From 601f4d6a925fc47a8f38a05cda85deb7f87fb44a Mon Sep 17 00:00:00 2001 From: comis-agent Date: Mon, 27 Jul 2026 14:22:48 +0300 Subject: [PATCH 20/33] feat(observability): add a bound-proof prefix-hash ladder to the cache trace MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit messageFingerprints is replaced wholesale by a bounded-payload sentinel once the array passes the persistence limit — at 247 messages it is already gone. That is exactly the regime where cache economics matter, so "where did the cached prefix diverge?" becomes unanswerable from the record built to answer it. Live case it blocks: a session whose cross-turn hit ratio fell from 98% at a ~120k-token prefix to 33% and then 0% as the conversation grew, with a stable systemDigest and a stable toolsDigest throughout. Something in the message array stops matching, and the per-message fingerprints needed to locate it are stripped. Record cumulative hashes over the leading fingerprints at exponentially spaced depths plus the full length. Comparing two calls localizes the first divergence to a range in ~log(n) values, and the ladder is fixed-size so the bound never touches it. --- .../src/cache-trace/stream-fn-wrapper.test.ts | 75 +++++++++++++++++++ .../src/cache-trace/stream-fn-wrapper.ts | 25 +++++++ .../observability/src/cache-trace/types.ts | 3 + 3 files changed, 103 insertions(+) diff --git a/packages/observability/src/cache-trace/stream-fn-wrapper.test.ts b/packages/observability/src/cache-trace/stream-fn-wrapper.test.ts index 5daef3b17a..26852610d4 100644 --- a/packages/observability/src/cache-trace/stream-fn-wrapper.test.ts +++ b/packages/observability/src/cache-trace/stream-fn-wrapper.test.ts @@ -454,3 +454,78 @@ describe("buildCacheTraceWrapper toolsDigest", () => { expect(rec.toolCount).toBe(0); }); }); + +// --------------------------------------------------------------------------- +// Prefix-hash ladder +// --------------------------------------------------------------------------- + +/** + * messageFingerprints is replaced wholesale by a bounded-payload sentinel once + * the array grows past the persistence limit — which is exactly when cache + * economics matter. Locating WHERE two calls' cached prefixes diverge then + * becomes impossible from the trace. + * + * The ladder is a handful of cumulative hashes over the leading fingerprints at + * exponentially spaced depths. Comparing two calls localizes the first + * divergence to a range in ~log(n) values, and the payload stays fixed-size so + * the bound never strips it. + * + * Observed live: a session whose cross-turn cache hit ratio fell from 98% at a + * ~120k-token prefix to 0% at ~170k, with messageFingerprints unavailable at + * 247 messages. + */ +describe("buildCacheTraceWrapper prefix-hash ladder", () => { + function ctxWithMessages(n: number, mutateAt?: number): unknown { + const messages = Array.from({ length: n }, (_, i) => ({ + role: i % 2 === 0 ? "user" : "assistant", + content: mutateAt === i ? "CHANGED" : `m${i}`, + })); + return { messages, systemPrompt: "sys", tools: [] }; + } + + async function ladderFor(n: number, mutateAt?: number): Promise> { + const filePath = join(tmpDir, `ladder-${n}-${mutateAt ?? "none"}.jsonl`); + const trace = makeTrace({ includeMessages: false, filePath }); + const next: StreamFn = ((..._args: unknown[]) => + ({ usage: { cacheRead: 0, cacheWrite: 0 } }) as unknown as ReturnType) as StreamFn; + buildCacheTraceWrapper(trace)(next)( + fakeModel() as Parameters[0], + ctxWithMessages(n, mutateAt) as Parameters[1], + ); + await trace.flush(); + return readLines(filePath).find((l) => l.stage === "stream:context")!; + } + + it("emits a bounded ladder of cumulative prefix hashes", async () => { + const rec = await ladderFor(100); + const ladder = rec.messagePrefixHashes as Array<{ depth: number; hash: string }>; + expect(Array.isArray(ladder)).toBe(true); + expect(ladder.length).toBeGreaterThan(2); + expect(ladder.length).toBeLessThanOrEqual(12); + expect(ladder[0]!.hash).toMatch(/^[0-9a-f]+$/); + }); + + it("keeps the ladder identical for identical message arrays", async () => { + const a = await ladderFor(100); + const b = await ladderFor(100); + expect(a.messagePrefixHashes).toEqual(b.messagePrefixHashes); + }); + + it("localizes a late divergence to the deeper rungs only", async () => { + const clean = (await ladderFor(100)) as never; + const dirty = (await ladderFor(100, 90)) as never; + const A = (clean as { messagePrefixHashes: Array<{ depth: number; hash: string }> }).messagePrefixHashes; + const B = (dirty as { messagePrefixHashes: Array<{ depth: number; hash: string }> }).messagePrefixHashes; + // Rungs at depth <= 90 cover only unchanged messages and must match. + for (let i = 0; i < A.length; i++) { + if (A[i]!.depth <= 90) expect(B[i]!.hash, `depth ${A[i]!.depth}`).toBe(A[i]!.hash); + } + // At least one deeper rung must differ, or the ladder localizes nothing. + expect(A.some((r, i) => r.depth > 90 && B[i]!.hash !== r.hash)).toBe(true); + }); + + it("survives the array bound that strips messageFingerprints", async () => { + const rec = await ladderFor(400); + expect(Array.isArray(rec.messagePrefixHashes)).toBe(true); + }); +}); diff --git a/packages/observability/src/cache-trace/stream-fn-wrapper.ts b/packages/observability/src/cache-trace/stream-fn-wrapper.ts index 231593af12..939a263c9c 100644 --- a/packages/observability/src/cache-trace/stream-fn-wrapper.ts +++ b/packages/observability/src/cache-trace/stream-fn-wrapper.ts @@ -214,6 +214,30 @@ function computeAssembledShape( * should not call this when `createCacheTrace` returned * null). */ +/** + * Cumulative hashes over the leading message fingerprints at exponentially + * spaced depths, plus the full length. + * + * `messageFingerprints` is replaced by a bounded-payload sentinel once the array + * grows past the persistence limit — precisely in the long sessions where cache + * economics matter — which makes "where did the cached prefix diverge?" + * unanswerable from the trace. Comparing two calls' ladders localizes the first + * divergence to a range in ~log(n) values, and the payload is fixed-size so the + * bound never strips it. + */ +function buildPrefixHashLadder( + fingerprints: readonly string[], +): Array<{ depth: number; hash: string }> { + const ladder: Array<{ depth: number; hash: string }> = []; + const depths: number[] = []; + for (let d = 8; d < fingerprints.length; d *= 2) depths.push(d); + if (fingerprints.length > 0) depths.push(fingerprints.length); + for (const depth of depths) { + ladder.push({ depth, hash: sha256(fingerprints.slice(0, depth).join("|")).slice(0, 16) }); + } + return ladder; +} + export function buildCacheTraceWrapper(trace: CacheTrace): StreamFnWrapper { return function cacheTraceWrapper(next: StreamFn): StreamFn { return (( @@ -256,6 +280,7 @@ export function buildCacheTraceWrapper(trace: CacheTrace): StreamFnWrapper { systemDigest, toolCount: tools.length, toolsDigest, + messagePrefixHashes: buildPrefixHashLadder(messageFingerprints), }; // The SMALL assembled-array shape descriptor (counts/flags + tool // id pairing). Emitted OUTSIDE the includeMessages guard so it is diff --git a/packages/observability/src/cache-trace/types.ts b/packages/observability/src/cache-trace/types.ts index 329385966c..48ed855bd0 100644 --- a/packages/observability/src/cache-trace/types.ts +++ b/packages/observability/src/cache-trace/types.ts @@ -155,6 +155,9 @@ export const CacheTraceEventSchema = z.object({ // The tool array — the prefix component between system and messages. A // change here invalidates every cached block after the system prompt. toolCount: z.number().int().nonnegative().optional(), + // Cumulative prefix hashes at exponentially spaced depths — fixed-size, so it + // survives the array bound that strips messageFingerprints on long sessions. + messagePrefixHashes: z.array(z.object({ depth: z.number().int(), hash: z.string() })).optional(), toolsDigest: z.string().optional(), // Token attribution. `session:after` aggregates across the session via // the EventBus bridge stash; `model:after` carries the per-call snapshot From f23c6ddfbb4d15956e5f64fded5aefdef2a35148 Mon Sep 17 00:00:00 2001 From: comis-agent Date: Mon, 27 Jul 2026 15:15:29 +0300 Subject: [PATCH 21/33] fix(agent): stop the lcd-evict log claiming eviction that did not happen MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The DEBUG line read "lcd history evicted under budget" on every assembly, including when droppedCount was 0 and nothing had been dropped. During a live cache investigation that reads as a statement that the message array was rewritten — the first thing to rule out when a cached prefix stops matching — and it cost a full detour down the wrong layer before droppedCount turned out to be 0 on every call of the run. State what happened: eviction only when something was evicted, otherwise that the history fit. The structured fields are unchanged. --- .../agent/src/context-engine/lcd-assembler.ts | 17 ++++++++++++- .../lcd-budget-eviction.test.ts | 25 +++++++++++++++++++ 2 files changed, 41 insertions(+), 1 deletion(-) diff --git a/packages/agent/src/context-engine/lcd-assembler.ts b/packages/agent/src/context-engine/lcd-assembler.ts index 90cd06fe04..59470e3ffe 100644 --- a/packages/agent/src/context-engine/lcd-assembler.ts +++ b/packages/agent/src/context-engine/lcd-assembler.ts @@ -58,6 +58,21 @@ import { partsToMessage } from "@comis/core"; import { neutralizeForgedMarkersInMessage } from "../session/forged-context-markers.js"; + +/** + * Message for the lcd-evict DEBUG. + * + * The line used to read "lcd history evicted under budget" unconditionally, + * including when droppedCount was 0. During a live cache investigation that + * reads as "the message array was rewritten" — the first thing to rule out when + * a cached prefix stops matching — and cost a full diagnostic detour before + * droppedCount showed 0 on every call. State what actually happened. + */ +export function evictionLogMessage(droppedCount: number): string { + return droppedCount > 0 + ? "lcd history evicted under budget" + : "lcd history fit under budget (no eviction)"; +} import type { ContextStorePort, ContextStoreScope, @@ -523,7 +538,7 @@ export function createLcdContextEngine( agentId: deps.agentId, sessionKey: deps.sessionKey, }, - "lcd history evicted under budget", + evictionLogMessage(droppedCount), ); // Emit the content-free `context:evicted` event (parity with the pipeline engine) diff --git a/packages/agent/src/context-engine/lcd-budget-eviction.test.ts b/packages/agent/src/context-engine/lcd-budget-eviction.test.ts index 97dff66590..3496d91790 100644 --- a/packages/agent/src/context-engine/lcd-budget-eviction.test.ts +++ b/packages/agent/src/context-engine/lcd-budget-eviction.test.ts @@ -26,6 +26,7 @@ import type { Message } from "@earendil-works/pi-ai"; import type { AgentMessage } from "@earendil-works/pi-agent-core"; import { describe, it, expect } from "vitest"; import { evictHistoryUnderBudget } from "./lcd-budget-eviction.js"; +import { evictionLogMessage } from "./lcd-assembler.js"; // --------------------------------------------------------------------------- // Fixtures (mirrors lcd-assembler.test.ts message shapes) @@ -262,3 +263,27 @@ describe("evictHistoryUnderBudget", () => { expect(out.map(textOf)).toEqual(["a1", "a2"]); }); }); + +// --------------------------------------------------------------------------- +// Eviction log honesty +// --------------------------------------------------------------------------- + +/** + * The lcd-evict DEBUG fired on every assembly with the message "lcd history + * evicted under budget" even when droppedCount was 0 — i.e. when nothing was + * evicted at all. Reading it during a live cache investigation, it is a + * statement that the message array was rewritten, which is the single most + * important thing to rule out when a cached prefix stops matching. It cost a + * full diagnostic detour before droppedCount showed 0 on every call. + * + * The line must state what happened, so the two cases read differently. + */ +describe("lcd-evict log honesty", () => { + it("does not claim eviction when nothing was dropped", () => { + expect(evictionLogMessage(0)).not.toMatch(/evicted/i); + }); + + it("states eviction when messages were actually dropped", () => { + expect(evictionLogMessage(3)).toMatch(/evicted/i); + }); +}); From f3d8fe2b5cc6fc62c8da99c1fc384996b8358afd Mon Sep 17 00:00:00 2001 From: comis-agent Date: Mon, 27 Jul 2026 16:53:14 +0300 Subject: [PATCH 22/33] fix(agent): no dangling em-dash when the failed tool cannot be named MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The tool-failure notice ends in an em-dash because the caller appends the failing tool's name verbatim. When the ONLY unrecovered failure is the background poller — which must never be named, since it merely relays another tool's failure — the lookup returns undefined and appended an empty string, so the user's reply ended: "...so part of this may be incomplete — " Observed at the end of a Hebrew answer about a report tool that hit the MCP call deadline: a truncated sentence trailing off into nothing, in the one place the turn was trying to be honest about being degraded. Add a complete-sentence variant to the locale set and select on whether a name exists. Regression introduced when the poller was excluded from blame. --- .../agent/src/executor/degraded-reply-i18n.ts | 19 +++++++++++- .../agent/src/executor/degraded-reply.test.ts | 31 +++++++++++++++++++ packages/agent/src/executor/degraded-reply.ts | 13 ++++++++ .../src/executor/executor-post-execution.ts | 11 +++++-- 4 files changed, 70 insertions(+), 4 deletions(-) diff --git a/packages/agent/src/executor/degraded-reply-i18n.ts b/packages/agent/src/executor/degraded-reply-i18n.ts index eabb191df4..303f3e220c 100644 --- a/packages/agent/src/executor/degraded-reply-i18n.ts +++ b/packages/agent/src/executor/degraded-reply-i18n.ts @@ -18,7 +18,8 @@ export type LocaleMessageId = | "output_starved" | "loop_detected" | "pipeline_timeout" - | "tool_failure_notice"; + | "tool_failure_notice" + | "tool_failure_notice_unnamed"; export type LocalePack = Readonly>>; @@ -49,6 +50,9 @@ const ENGLISH_PACK: Readonly> = { tool_failure_notice: "\n\nNote: one of the tools I used reported an error, so part of this may be" + " incomplete — ", + tool_failure_notice_unnamed: + "\n\nNote: one of the tools I used reported an error, so part of this may be" + + " incomplete.", pipeline_timeout: "I stopped this request because it was taking too long and hit the time limit " + "for a single turn. Nothing was left half-applied. If it needs many lookups, " @@ -198,6 +202,19 @@ export function selectToolFailureNotice( return catalog.resolve(locale, "tool_failure_notice"); } +/** + * The tool-failure notice for the case with NO nameable culprit — the only + * unrecovered failure was the background poller, which relays other tools' + * failures and must never be blamed. The named variant ends in an em-dash + * awaiting a tool name; using it here left the reply ending "incomplete — ". + */ +export function selectToolFailureNoticeUnnamed( + locale: string | undefined, + catalog: LocaleCatalog = DEFAULT_LOCALE_CATALOG, +): string { + return catalog.resolve(locale, "tool_failure_notice_unnamed"); +} + export function selectPipelineTimeoutReply( locale: string | undefined, opts: { traceId?: string }, diff --git a/packages/agent/src/executor/degraded-reply.test.ts b/packages/agent/src/executor/degraded-reply.test.ts index f0f4357713..40695a1911 100644 --- a/packages/agent/src/executor/degraded-reply.test.ts +++ b/packages/agent/src/executor/degraded-reply.test.ts @@ -15,6 +15,8 @@ import { describe, it, expect } from "vitest"; import { buildOutputStarvedAnnotation, + buildToolFailureNotice, + buildToolFailureNoticeUnnamed, buildContextExhaustedReply, buildLoopDetectedReply, buildDegradedReply, @@ -297,3 +299,32 @@ describe("builders consume the resolved language tag (delegate to i18n)", () => ); }); }); + +// --------------------------------------------------------------------------- +// Nameless tool-failure notice +// --------------------------------------------------------------------------- + +/** + * The named notice ends with an em-dash because the caller appends the failing + * tool's name verbatim. When the only unrecovered failure is the background + * poller — which must never be named as the culprit, since it merely relays + * another tool's failure — there is no name to append, and the reply ended in a + * dangling "incomplete — " with nothing after it. + * + * Seen live at the end of a Hebrew answer about a timed-out report tool. + */ +describe("buildToolFailureNoticeUnnamed", () => { + it("ends as a complete sentence with no dangling dash", () => { + const text = buildToolFailureNoticeUnnamed(); + expect(text.trimEnd()).not.toMatch(/[—-]$/); + expect(text.trimEnd()).toMatch(/\.$/); + }); + + it("still separates itself from the preceding reply", () => { + expect(buildToolFailureNoticeUnnamed().startsWith("\n\n")).toBe(true); + }); + + it("differs from the named variant", () => { + expect(buildToolFailureNoticeUnnamed()).not.toBe(buildToolFailureNotice()); + }); +}); diff --git a/packages/agent/src/executor/degraded-reply.ts b/packages/agent/src/executor/degraded-reply.ts index f819e6239f..ed25d6b70c 100644 --- a/packages/agent/src/executor/degraded-reply.ts +++ b/packages/agent/src/executor/degraded-reply.ts @@ -25,6 +25,7 @@ import { selectLoopDetectedReply, selectPipelineTimeoutReply, selectToolFailureNotice, + selectToolFailureNoticeUnnamed, type LocaleCatalog, } from "./degraded-reply-i18n.js"; @@ -136,3 +137,15 @@ export function buildToolFailureNotice( ): string { return selectToolFailureNotice(language, localeCatalog); } + +/** + * Localized tool-failure notice for the case with no nameable culprit. Reads as + * a complete sentence — the named variant deliberately ends in an em-dash so the + * caller can append the tool name. + */ +export function buildToolFailureNoticeUnnamed( + language?: string, + localeCatalog?: LocaleCatalog, +): string { + return selectToolFailureNoticeUnnamed(language, localeCatalog); +} diff --git a/packages/agent/src/executor/executor-post-execution.ts b/packages/agent/src/executor/executor-post-execution.ts index d3dafd939c..12b6274d3b 100644 --- a/packages/agent/src/executor/executor-post-execution.ts +++ b/packages/agent/src/executor/executor-post-execution.ts @@ -121,7 +121,7 @@ import { createHash, randomUUID } from "node:crypto"; // Critic hook (no inline logic — all logic in verification-gate.ts) import { shouldRunCritic, runVerificationCritic } from "./verification-gate.js"; // Deterministic user-facing replies for named degraded terminal causes. -import { buildOutputStarvedAnnotation, buildContextExhaustedReply, buildLoopDetectedReply, buildToolFailureNotice, catalogFromLocalePacks, LOCALE_MESSAGE_IDS } from "./degraded-reply.js"; +import { buildOutputStarvedAnnotation, buildContextExhaustedReply, buildLoopDetectedReply, buildToolFailureNotice, buildToolFailureNoticeUnnamed, catalogFromLocalePacks, LOCALE_MESSAGE_IDS } from "./degraded-reply.js"; import { BACKGROUND_POLLER_TOOL } from "../safety/background-failure-attribution.js"; import { parseContextExhaustionCause } from "../context-engine/errors.js"; import { buildSyntheticCriticDeps } from "./verification-gate-synth-deps.js"; @@ -1410,9 +1410,14 @@ export async function postExecution(params: PostExecutionParams): Promise // No "(see session log)" pointer: the recipient is the CHAT user, who has // no session log to see — the operator's lens is `comis explain` (the // failure rides the trajectory + IncidentReport.failures already). + // Two variants: the named notice ends in an em-dash awaiting the tool name; + // the unnamed one is a complete sentence. Using the named form with no name + // left replies ending in a dangling "incomplete — " whenever the poller was + // the only unrecovered failure. result.response = (result.response ?? "") - + buildToolFailureNotice(replyLanguage, localeCatalog) - + (failedToolName ?? ""); + + (failedToolName === undefined + ? buildToolFailureNoticeUnnamed(replyLanguage, localeCatalog) + : buildToolFailureNotice(replyLanguage, localeCatalog) + failedToolName); } // Degrade loudly — deliver an honest user-facing reply for named degraded causes. From d37ce35bbf3afacecd7474c3ce3717e6923e9ba3 Mon Sep 17 00:00:00 2001 From: comis-agent Date: Mon, 27 Jul 2026 17:37:08 +0300 Subject: [PATCH 23/33] fix(agent): localize the stall/whole-turn timeout reply MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The prompt_timeout userMessage was a hard-coded English literal in error-classifier.ts, delivered verbatim into conversations in any language. It is the same gap already closed for pipeline_timeout, and it matters for the same reason: a stalled turn is one of the few messages a user is guaranteed to see, so it is exactly the one that must sit inside the localizable platform-reply set rather than beside it. Observed live: a Hebrew conversation stalled for 404s and the user received "The request took too long to process. Please try again with a simpler message." Add prompt_timeout to the locale message set and select it at the delivery site in failure-path.ts. Other classifier categories keep their literal text until they are given ids too — this changes only the category that was observed. --- .../agent/src/executor/degraded-reply-i18n.ts | 17 ++++++++- .../agent/src/executor/degraded-reply.test.ts | 37 +++++++++++++++++++ packages/agent/src/executor/degraded-reply.ts | 12 ++++++ .../executor/prompt-runner/failure-path.ts | 15 +++++++- 4 files changed, 79 insertions(+), 2 deletions(-) diff --git a/packages/agent/src/executor/degraded-reply-i18n.ts b/packages/agent/src/executor/degraded-reply-i18n.ts index 303f3e220c..63afdeda27 100644 --- a/packages/agent/src/executor/degraded-reply-i18n.ts +++ b/packages/agent/src/executor/degraded-reply-i18n.ts @@ -19,7 +19,8 @@ export type LocaleMessageId = | "loop_detected" | "pipeline_timeout" | "tool_failure_notice" - | "tool_failure_notice_unnamed"; + | "tool_failure_notice_unnamed" + | "prompt_timeout"; export type LocalePack = Readonly>>; @@ -53,6 +54,8 @@ const ENGLISH_PACK: Readonly> = { tool_failure_notice_unnamed: "\n\nNote: one of the tools I used reported an error, so part of this may be" + " incomplete.", + prompt_timeout: + "The request took too long to process. Please try again with a simpler message.", pipeline_timeout: "I stopped this request because it was taking too long and hit the time limit " + "for a single turn. Nothing was left half-applied. If it needs many lookups, " @@ -215,6 +218,18 @@ export function selectToolFailureNoticeUnnamed( return catalog.resolve(locale, "tool_failure_notice_unnamed"); } +/** + * The reply for a turn killed by the stall budget or whole-turn retry timeout. + * Was a hard-coded English literal in error-classifier.ts, shipped verbatim into + * conversations in any language. + */ +export function selectPromptTimeoutReply( + locale: string | undefined, + catalog: LocaleCatalog = DEFAULT_LOCALE_CATALOG, +): string { + return catalog.resolve(locale, "prompt_timeout"); +} + export function selectPipelineTimeoutReply( locale: string | undefined, opts: { traceId?: string }, diff --git a/packages/agent/src/executor/degraded-reply.test.ts b/packages/agent/src/executor/degraded-reply.test.ts index 40695a1911..bed2334120 100644 --- a/packages/agent/src/executor/degraded-reply.test.ts +++ b/packages/agent/src/executor/degraded-reply.test.ts @@ -16,10 +16,13 @@ import { describe, it, expect } from "vitest"; import { buildOutputStarvedAnnotation, buildToolFailureNotice, + buildPromptTimeoutReply, buildToolFailureNoticeUnnamed, buildContextExhaustedReply, buildLoopDetectedReply, buildDegradedReply, + catalogFromLocalePacks, + LOCALE_MESSAGE_IDS, } from "./degraded-reply.js"; import { selectOutputStarvedAnnotation, @@ -328,3 +331,37 @@ describe("buildToolFailureNoticeUnnamed", () => { expect(buildToolFailureNoticeUnnamed()).not.toBe(buildToolFailureNotice()); }); }); + +// --------------------------------------------------------------------------- +// Prompt-timeout reply +// --------------------------------------------------------------------------- + +/** + * The whole-turn / stall-budget timeout reply was a hard-coded English literal + * in error-classifier.ts, delivered verbatim regardless of the conversation's + * language — the same gap already closed for pipeline_timeout. A stalled turn is + * one of the few messages a user is guaranteed to see, so it is exactly the one + * that must live inside the localizable platform-reply set. + * + * Observed live: a Hebrew conversation whose 404s stall produced + * "The request took too long to process. Please try again with a simpler message." + */ +describe("buildPromptTimeoutReply", () => { + it("returns the canonical English text with no locale configured", () => { + expect(buildPromptTimeoutReply()).toContain("took too long"); + }); + + it("is a member of the locale message set", () => { + expect(LOCALE_MESSAGE_IDS).toContain("prompt_timeout"); + }); + + it("uses an operator-supplied pack for the resolved locale", () => { + const catalog = catalogFromLocalePacks({ he: { prompt_timeout: "לקח יותר מדי זמן" } }); + expect(buildPromptTimeoutReply("he", catalog)).toBe("לקח יותר מדי זמן"); + }); + + it("falls back to English for a locale with no pack", () => { + const catalog = catalogFromLocalePacks({ he: { prompt_timeout: "x" } }); + expect(buildPromptTimeoutReply("fr", catalog)).toContain("took too long"); + }); +}); diff --git a/packages/agent/src/executor/degraded-reply.ts b/packages/agent/src/executor/degraded-reply.ts index ed25d6b70c..8d6516157d 100644 --- a/packages/agent/src/executor/degraded-reply.ts +++ b/packages/agent/src/executor/degraded-reply.ts @@ -26,6 +26,7 @@ import { selectPipelineTimeoutReply, selectToolFailureNotice, selectToolFailureNoticeUnnamed, + selectPromptTimeoutReply, type LocaleCatalog, } from "./degraded-reply-i18n.js"; @@ -149,3 +150,14 @@ export function buildToolFailureNoticeUnnamed( ): string { return selectToolFailureNoticeUnnamed(language, localeCatalog); } + +/** + * Localized reply for a turn killed by the stall budget or the whole-turn retry + * timeout. PURE: same input -> same string. + */ +export function buildPromptTimeoutReply( + language?: string, + localeCatalog?: LocaleCatalog, +): string { + return selectPromptTimeoutReply(language, localeCatalog); +} diff --git a/packages/agent/src/executor/prompt-runner/failure-path.ts b/packages/agent/src/executor/prompt-runner/failure-path.ts index ed7e8e1e02..76bfdb7c45 100644 --- a/packages/agent/src/executor/prompt-runner/failure-path.ts +++ b/packages/agent/src/executor/prompt-runner/failure-path.ts @@ -14,6 +14,7 @@ * @module */ +import { buildPromptTimeoutReply, catalogFromLocalePacks } from "../degraded-reply.js"; import { formatSessionKey, resolveModelPricing, scriptTokenFactor } from "@comis/core"; import type { ErrorKind } from "@comis/core"; @@ -174,7 +175,19 @@ function emitFailureDiagnostics( if (classified.category === "auth_invalid") { result.response = `The AI service could not authenticate with the "${config.provider}" provider. Please check the API key or notify the system administrator.`; } else { - result.response = classified.userMessage; + // Route the stall/whole-turn timeout through the locale seam. It was a + // hard-coded English literal in error-classifier.ts, delivered verbatim into + // conversations in any language — and a stalled turn is one of the few + // messages a user is guaranteed to see. Other categories keep their + // classifier text until they are given locale ids too. + result.response = classified.category === "prompt_timeout" + ? buildPromptTimeoutReply( + (config as { language?: string }).language, + catalogFromLocalePacks( + (config as { localePacks?: Record> }).localePacks, + ), + ) + : classified.userMessage; } result.errorContext = { errorType: isPromptTimeout ? "PromptTimeout" : "PromptFailure", From 08e26f82464e142e6069a8f4b17a6766094067d0 Mon Sep 17 00:00:00 2001 From: comis-agent Date: Mon, 27 Jul 2026 18:26:26 +0300 Subject: [PATCH 24/33] fix(skills): stop erasing composed MCP schemas into Type.Any() MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit anyOf / oneOf / allOf reached the Type.Any() fallback, which removes the type entirely: the model is told nothing about the parameter's shape AND local validation accepts any value, so a wrong type travels to the MCP server and returns as an opaque -32602. Same silent-information-loss class as the numeric bounds and enums that were being stripped. A composed schema frequently carries no top-level `type`, so the branch has to run BEFORE the type checks or it is unreachable. Observed live: a model passed an object-typed parameter as a string, local validation waved it through, and the server rejected it twice in one turn. Whether that specific parameter arrives as a composed schema is not confirmed — the fidelity gap is real and worth closing on its own terms, so this is not claimed as the fix for that turn. $ref still falls back to Type.Any(): resolving it needs the document root, which this function does not receive. --- .../src/skills/bridge/mcp-tool-bridge.test.ts | 58 +++++++++++++++++++ .../src/skills/bridge/mcp-tool-bridge.ts | 21 ++++++- 2 files changed, 77 insertions(+), 2 deletions(-) diff --git a/packages/skills/src/skills/bridge/mcp-tool-bridge.test.ts b/packages/skills/src/skills/bridge/mcp-tool-bridge.test.ts index c65b34a791..3d078d356f 100644 --- a/packages/skills/src/skills/bridge/mcp-tool-bridge.test.ts +++ b/packages/skills/src/skills/bridge/mcp-tool-bridge.test.ts @@ -908,3 +908,61 @@ describe("mcpToolsToAgentTools - wrapExternalContent integration", () => { expect(text).toMatch(/<<>>/); }); }); + +// --------------------------------------------------------------------------- +// Composed schemas +// --------------------------------------------------------------------------- + +/** + * anyOf / oneOf / allOf collapsed to Type.Any(), which erases the type entirely: + * the model is told nothing about the shape, and local validation accepts any + * value — so a wrong type travels all the way to the MCP server and comes back + * as an opaque -32602. That is the same silent-information-loss class as the + * stripped numeric bounds and enums above. + * + * Observed live: a model passed an object-typed parameter as a string, local + * validation waved it through, and the server rejected it twice. + */ +describe("jsonSchemaToTypeBox preserves composed schemas", () => { + it("converts anyOf into a union that still rejects a wrong type", () => { + const schema = jsonSchemaToTypeBox({ + anyOf: [{ type: "object", properties: { a: { type: "string" } } }, { type: "null" }], + }); + // A union, not an untyped Any — Any has no discriminating keywords at all. + expect(JSON.stringify(schema)).toMatch(/anyOf|oneOf/); + }); + + it("converts oneOf into a union", () => { + const schema = jsonSchemaToTypeBox({ + oneOf: [{ type: "string" }, { type: "number" }], + }); + expect(JSON.stringify(schema)).toMatch(/anyOf|oneOf/); + }); + + it("keeps the object branch typed inside anyOf", () => { + const schema = jsonSchemaToTypeBox({ + anyOf: [{ type: "object", properties: { from_date: { type: "string" } } }], + }); + expect(JSON.stringify(schema)).toContain("from_date"); + }); + + it("carries a description through a composed schema", () => { + const schema = jsonSchemaToTypeBox({ + description: "the second window", + anyOf: [{ type: "string" }, { type: "number" }], + }); + expect(JSON.stringify(schema)).toContain("the second window"); + }); + + it("converts allOf into an intersection", () => { + const schema = jsonSchemaToTypeBox({ + allOf: [ + { type: "object", properties: { a: { type: "string" } } }, + { type: "object", properties: { b: { type: "number" } } }, + ], + }); + const s = JSON.stringify(schema); + expect(s).toContain("a"); + expect(s).toContain("b"); + }); +}); diff --git a/packages/skills/src/skills/bridge/mcp-tool-bridge.ts b/packages/skills/src/skills/bridge/mcp-tool-bridge.ts index 5f4c4faa71..cb99b90a53 100644 --- a/packages/skills/src/skills/bridge/mcp-tool-bridge.ts +++ b/packages/skills/src/skills/bridge/mcp-tool-bridge.ts @@ -59,8 +59,9 @@ export function classifyMcpErrorType(errorText: string | undefined): string { /** * Convert a basic JSON Schema definition to a TypeBox TSchema. * - * Handles primitive types, arrays, and objects. Complex schema features - * (oneOf, allOf, $ref, etc.) fall back to Type.Any(). + * Handles primitive types, arrays, objects, and the composition keywords + * anyOf / oneOf / allOf. `$ref` still falls back to Type.Any() — resolving it + * needs the document root, which this function does not receive. * * This is intentionally simple -- MCP tool schemas are typically flat * objects with primitive properties. Complex schemas still work but @@ -70,6 +71,22 @@ export function jsonSchemaToTypeBox(schema: Record): TSchema { const type = schema.type; const annotations = carriedKeywords(schema); + // Composition keywords, BEFORE the `type` checks — a composed schema often + // carries no top-level `type` at all, so it used to reach the Type.Any() + // fallback and erase the shape completely: the model was told nothing, local + // validation accepted any value, and a wrong type only surfaced as an opaque + // MCP -32602 from the server. Live: an object-typed parameter sent as a + // string, waved through locally, rejected twice upstream. + const anyOf = schema.anyOf ?? schema.oneOf; + if (Array.isArray(anyOf) && anyOf.length > 0) { + const variants = (anyOf as Array>).map(jsonSchemaToTypeBox); + return Type.Union(variants, annotations); + } + if (Array.isArray(schema.allOf) && schema.allOf.length > 0) { + const parts = (schema.allOf as Array>).map(jsonSchemaToTypeBox); + return Type.Intersect(parts, annotations); + } + if (type === "string") { return Type.String(annotations); } From d0044d2434ddbdf1f9f2c095f2c1946d828c734f Mon Sep 17 00:00:00 2001 From: comis-agent Date: Mon, 27 Jul 2026 20:45:52 +0300 Subject: [PATCH 25/33] fix(skills): register the MCP progress handler on every path MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The SDK only accepts a progress notification when the request registered a handler for it. `onprogress` was gated on a trace context, so on any UNTRACED path — which is exactly where a backgrounded call runs — a server that reports progress produced: MCP client error: Received a progress notification for an unknown token and the client CLOSED THE CONNECTION, failing the tool with -32000 and forcing a reconnect mid-turn. This is the same mistake, in the same options object, that the ceiling above was already fixed for: `maxTotalTimeout` had been scoped to the trace branch and was made unconditional because a deadline that applies only when tracing happens to be on is not a deadline. The progress handler sat in that branch untouched, and the identical reasoning applies — the untraced background path is precisely where long, progress-reporting calls live. Observed live: a progress-reporting MCP tool failed with -32000 Connection closed, followed by "MCP server reconnect started". --- .../skills/integrations/mcp-client.test.ts | 46 ++++++++++++++++++- .../mcp-client/mcp-client-call.ts | 15 +++--- 2 files changed, 54 insertions(+), 7 deletions(-) diff --git a/packages/skills/src/skills/integrations/mcp-client.test.ts b/packages/skills/src/skills/integrations/mcp-client.test.ts index b671a3e6ac..4937849cd0 100644 --- a/packages/skills/src/skills/integrations/mcp-client.test.ts +++ b/packages/skills/src/skills/integrations/mcp-client.test.ts @@ -8,6 +8,7 @@ */ import { describe, it, expect, vi, beforeEach, afterEach } from "vitest"; +import { readFileSync } from "node:fs"; import type { TypedEventBus } from "@comis/core"; import { MCP_CALL_TOOL_TIMEOUT_MS_DEFAULT } from "@comis/core"; @@ -614,6 +615,8 @@ describe("McpClientManager", () => { { timeout: MCP_CALL_TOOL_TIMEOUT_MS_DEFAULT, maxTotalTimeout: MCP_CALL_TOOL_TIMEOUT_MS_DEFAULT, + onprogress: expect.any(Function), + resetTimeoutOnProgress: true, }, ); }); @@ -757,7 +760,15 @@ describe("McpClientManager", () => { expect(mockCallTool).toHaveBeenCalledWith( { name: "search", arguments: { query: "test" } }, undefined, - { timeout: 120_000, maxTotalTimeout: 120_000 }, + { + timeout: 120_000, + maxTotalTimeout: 120_000, + // Registered on EVERY path: without a handler the SDK rejects an + // incoming progress notification as an unknown token and closes the + // connection. + onprogress: expect.any(Function), + resetTimeoutOnProgress: true, + }, ); }); @@ -1764,3 +1775,36 @@ describe("keepalive queue routing (concurrency-aware)", () => { ); }); }); + +// --------------------------------------------------------------------------- +// Progress-handler registration +// --------------------------------------------------------------------------- + +/** + * The SDK only accepts a progress notification when the request registered a + * handler for it. Gating `onprogress` on a trace context meant that on ANY + * untraced path — which is exactly where a backgrounded call runs — a server + * that reports progress produced: + * + * MCP client error: Received a progress notification for an unknown token + * + * and the client CLOSED THE CONNECTION, failing the tool with -32000 and forcing + * a reconnect. The same reasoning already made `maxTotalTimeout` unconditional: + * a protection that applies only when tracing happens to be on is not a + * protection. + */ +describe("callTool registers a progress handler on every path", () => { + it("passes onprogress whether or not a trace context exists", () => { + const source = readFileSync( + new URL("./mcp-client/mcp-client-call.ts", import.meta.url), + "utf8", + ); + const optionsBlock = source.slice( + source.indexOf("maxTotalTimeout:"), + source.indexOf("maxTotalTimeout:") + 1200, + ); + // onprogress must NOT sit inside a requestTraceId conditional. + expect(optionsBlock).toContain("onprogress"); + expect(optionsBlock).not.toMatch(/requestTraceId\s*\?[\s\S]{0,120}onprogress/); + }); +}); diff --git a/packages/skills/src/skills/integrations/mcp-client/mcp-client-call.ts b/packages/skills/src/skills/integrations/mcp-client/mcp-client-call.ts index a64eb7d6f5..8055b3973a 100644 --- a/packages/skills/src/skills/integrations/mcp-client/mcp-client-call.ts +++ b/packages/skills/src/skills/integrations/mcp-client/mcp-client-call.ts @@ -223,12 +223,15 @@ export async function callTool( // long-running background call runs, kept no ceiling at all. A // deadline that applies only when tracing is on is not a deadline. maxTotalTimeout: state.options.callToolTimeoutMs, - ...(requestTraceId - ? { - onprogress: () => {}, - resetTimeoutOnProgress: true, - } - : {}), + // UNCONDITIONAL, for the same reason as the ceiling above. The SDK only + // accepts a progress notification when the request registered a handler + // for it; gating this on a trace context meant that on any UNTRACED + // path — which is exactly where a backgrounded call runs — a server + // that reports progress produced "Received a progress notification for + // an unknown token" and the client CLOSED THE CONNECTION, failing the + // tool with -32000 and forcing a reconnect. Live: a progress-reporting MCP tool on a background path. + onprogress: () => {}, + resetTimeoutOnProgress: true, }, ); From b79b45b34b49e4c066c9ec6d17936bc7052e481c Mon Sep 17 00:00:00 2001 From: comis-agent Date: Mon, 27 Jul 2026 22:32:53 +0300 Subject: [PATCH 26/33] fix(agent): deliver the delegation policy before the model has to guess it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit SYSTEM_PROMPT_GUIDES["sessions_spawn"] holds the "## Task Delegation" policy — the >30s rule, the media/multi-file/deep-research criteria, the parallel fan-out instruction — and jit-guide-injector delivered it ONLY after a successful sessions_spawn call. That is circular: the policy exists to make the model reach for the tool, and it was unreachable until the model had already reached for it. Moving the trigger into the always-present lean description was not enough. Across 119 live prompts on two different model families the agent delegated ZERO times, including an eleven-tool, multi-minute, file-producing turn that matches three of the criteria outright. A direct request proved the tool is present and works, so this was never missing plumbing — the model simply never learned the policy. Deliver it on the first tool result of ANY kind when sessions_spawn is on the surface. Keyed to the policy rather than the tool that carried it, so a later spawn does not repeat the whole section, and gated on availability so an agent without the tool is never told to use it. Callers passing no options are byte-identical to before. --- .../src/executor/jit-guide-injector.test.ts | 56 +++++++++++++++++++ .../agent/src/executor/jit-guide-injector.ts | 31 +++++++++- .../src/executor/pi-executor/pi-executor.ts | 9 ++- 3 files changed, 93 insertions(+), 3 deletions(-) diff --git a/packages/agent/src/executor/jit-guide-injector.test.ts b/packages/agent/src/executor/jit-guide-injector.test.ts index e5ef950565..4e2f8d5940 100644 --- a/packages/agent/src/executor/jit-guide-injector.test.ts +++ b/packages/agent/src/executor/jit-guide-injector.test.ts @@ -703,3 +703,59 @@ describe("same guide is not re-injected within a single turn", () => { expect(logger.info).toHaveBeenCalledTimes(1); }); }); + +// --------------------------------------------------------------------------- +// Delegation policy reachability +// --------------------------------------------------------------------------- + +/** + * SYSTEM_PROMPT_GUIDES["sessions_spawn"] holds the "## Task Delegation" policy — + * the >30s rule, the media/multi-file criteria, the parallel fan-out + * instruction — and it was delivered ONLY after a successful sessions_spawn call. + * That is circular: the policy exists to make the model reach for the tool, and + * it was unreachable until the model had already reached for it. + * + * Proven live: sessions_spawn is present and works (a direct request spawned a + * run), yet across 119 prompts on two different model families the agent never + * once delegated — including an 11-tool, multi-minute, file-producing turn that + * matches three of the criteria. + * + * Deliver it on the FIRST tool result of any kind when delegation is available, + * keyed so the later sessions_spawn call does not repeat it. + */ +describe("wrapToolResultWithGuide delivers the delegation policy early", () => { + function ok(): AgentToolResult { + return { content: [{ type: "text", text: "done" }], details: {} } as AgentToolResult; + } + + it("appends the delegation policy to an unrelated tool's first result", () => { + const delivered = new Set(); + const out = wrapToolResultWithGuide("read", ok(), delivered, createMockLogger(), { + delegationAvailable: true, + }); + expect(JSON.stringify(out.content)).toContain("Task Delegation"); + }); + + it("does not repeat it on a later sessions_spawn call", () => { + const delivered = new Set(); + wrapToolResultWithGuide("read", ok(), delivered, createMockLogger(), { delegationAvailable: true }); + const second = wrapToolResultWithGuide("sessions_spawn", ok(), delivered, createMockLogger(), { + delegationAvailable: true, + }); + expect(JSON.stringify(second.content)).not.toContain("## Task Delegation"); + }); + + it("stays silent when delegation is not available", () => { + const delivered = new Set(); + const out = wrapToolResultWithGuide("read", ok(), delivered, createMockLogger(), { + delegationAvailable: false, + }); + expect(JSON.stringify(out.content)).not.toContain("Task Delegation"); + }); + + it("is byte-identical to before when no options are passed", () => { + const delivered = new Set(); + const out = wrapToolResultWithGuide("read", ok(), delivered, createMockLogger()); + expect(JSON.stringify(out.content)).not.toContain("Task Delegation"); + }); +}); diff --git a/packages/agent/src/executor/jit-guide-injector.ts b/packages/agent/src/executor/jit-guide-injector.ts index 4a578d0129..6b4be0ba1e 100644 --- a/packages/agent/src/executor/jit-guide-injector.ts +++ b/packages/agent/src/executor/jit-guide-injector.ts @@ -67,11 +67,26 @@ const PRIVILEGED_SECTION_KEY = "section:privileged"; * the agent-loop on the message level), but some tools (MCP, discovery) * include it at runtime. We check for it defensively. */ +/** Options for the one-time guide injection. */ +export interface GuideInjectionOptions { + /** + * True when `sessions_spawn` is on this session's tool surface. + * + * The "## Task Delegation" policy is keyed on `sessions_spawn`, so it used to + * arrive only AFTER a successful spawn — circular, because the policy exists to + * make the model reach for the tool. Delivering it on the first tool result of + * ANY kind breaks that loop. Live: sessions_spawn is present and works, yet + * across 119 prompts on two model families the agent never delegated once. + */ + delegationAvailable?: boolean; +} + export function wrapToolResultWithGuide( toolName: string, result: AgentToolResult, deliveredGuides: Set, logger: ComisLogger, + options?: GuideInjectionOptions, ): AgentToolResult { // Two-phase design: first decide what WOULD fire without mutating state, // then commit (mark delivered + append) only if the result is non-error. @@ -86,8 +101,20 @@ export function wrapToolResultWithGuide( const toolGuide = getToolGuideWithSchema(toolName); const wantsTool = !!toolGuide && !deliveredGuides.has(toolName); - const sectionGuide = SYSTEM_PROMPT_GUIDES[toolName]; - const sectionKey = `section:${toolName}`; + // The delegation policy rides the FIRST tool result when spawning is available, + // not just a sessions_spawn result. Same delivery key, so a later spawn does + // not repeat it. + const delegationGuide = + options?.delegationAvailable === true && toolName !== "sessions_spawn" + ? SYSTEM_PROMPT_GUIDES["sessions_spawn"] + : undefined; + const sectionGuide = SYSTEM_PROMPT_GUIDES[toolName] ?? delegationGuide; + // Key the delegation policy to the POLICY, not the tool that happened to carry + // it — otherwise delivering it via `read` would not stop a later + // `sessions_spawn` call from repeating the whole section. + const sectionKey = delegationGuide !== undefined && SYSTEM_PROMPT_GUIDES[toolName] === undefined + ? "section:sessions_spawn" + : `section:${toolName}`; const wantsSection = !!sectionGuide && !deliveredGuides.has(sectionKey); const wantsPrivileged = diff --git a/packages/agent/src/executor/pi-executor/pi-executor.ts b/packages/agent/src/executor/pi-executor/pi-executor.ts index fdc8ab8498..2f09cd3290 100644 --- a/packages/agent/src/executor/pi-executor/pi-executor.ts +++ b/packages/agent/src/executor/pi-executor/pi-executor.ts @@ -1713,7 +1713,14 @@ async function runSessionLocked( onUpdate as Parameters[3], undefined as unknown as Parameters[4], ); - return wrapToolResultWithGuide(original.name, res, deliveredGuides, deps.logger); + return wrapToolResultWithGuide(original.name, res, deliveredGuides, deps.logger, { + // Any first tool result carries the delegation policy when spawning is + // reachable, so the model does not have to spawn in order to learn + // that it can. + delegationAvailable: contextTools.some( + (t) => (t as { name?: string }).name === "sessions_spawn", + ), + }); }, } as unknown as (typeof contextTools)[0]); injectedCount++; From 314333928405723545036a591b0a5db86ca37987 Mon Sep 17 00:00:00 2001 From: comis-agent Date: Mon, 27 Jul 2026 22:49:30 +0300 Subject: [PATCH 27/33] fix(agent): gate the delegation policy on the session tool list MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The gate added in the previous commit read `contextTools` at the discovery re-injection call site. That array holds only NEWLY DISCOVERED tools, so sessions_spawn was never in it and delegationAvailable was always false — the policy still never shipped. Verified live: the guide log showed only "tool:read", never "section:sessions_spawn". createJitGuideWrapper is the site that matters. It wraps the session's initial tools and is handed the full active tool array, so the availability check belongs there and is computed once per session rather than per tool call. Also settles what was suspected for a long stretch: sessions_spawn is NOT deferred. The only deferred tools on this deployment are the three channel actions (discord/slack/whatsapp). The tool has been on the surface the whole time, and a direct request spawns a run successfully — the model simply never learned the policy, because the policy was keyed on having already used it. --- .../src/executor/jit-guide-injector.test.ts | 33 +++++++++++++++++++ .../agent/src/executor/jit-guide-injector.ts | 9 ++++- 2 files changed, 41 insertions(+), 1 deletion(-) diff --git a/packages/agent/src/executor/jit-guide-injector.test.ts b/packages/agent/src/executor/jit-guide-injector.test.ts index 4e2f8d5940..549cbe2517 100644 --- a/packages/agent/src/executor/jit-guide-injector.test.ts +++ b/packages/agent/src/executor/jit-guide-injector.test.ts @@ -759,3 +759,36 @@ describe("wrapToolResultWithGuide delivers the delegation policy early", () => { expect(JSON.stringify(out.content)).not.toContain("Task Delegation"); }); }); + +/** + * The gate must read the SESSION tool list. An earlier version read the + * discovery-injection array at the other call site, which holds only newly + * discovered tools, so it was always false and the policy never shipped — + * verified live by a guide log showing only "tool:read". + */ +describe("createJitGuideWrapper gates delegation on the session tool list", () => { + function tool(name: string): ToolDefinition { + return { + name, + description: "", + parameters: {}, + execute: async () => ({ content: [{ type: "text", text: "ok" }], details: {} }), + } as unknown as ToolDefinition; + } + + it("ships the delegation policy when sessions_spawn is in the session tools", async () => { + const delivered = new Set(); + const [wrapped] = createJitGuideWrapper( + [tool("read"), tool("sessions_spawn")], delivered, createMockLogger(), + ); + const out = await wrapped!.execute("c1", {}, undefined, undefined, {} as never); + expect(JSON.stringify(out.content)).toContain("Task Delegation"); + }); + + it("stays silent when sessions_spawn is absent", async () => { + const delivered = new Set(); + const [wrapped] = createJitGuideWrapper([tool("read")], delivered, createMockLogger()); + const out = await wrapped!.execute("c1", {}, undefined, undefined, {} as never); + expect(JSON.stringify(out.content)).not.toContain("Task Delegation"); + }); +}); diff --git a/packages/agent/src/executor/jit-guide-injector.ts b/packages/agent/src/executor/jit-guide-injector.ts index 6b4be0ba1e..c639da662e 100644 --- a/packages/agent/src/executor/jit-guide-injector.ts +++ b/packages/agent/src/executor/jit-guide-injector.ts @@ -190,6 +190,11 @@ export function createJitGuideWrapper( deliveredGuides: Set, logger: ComisLogger, ): ToolDefinition[] { + // Computed ONCE from the full session tool list — the array this wrapper is + // handed IS the active surface, unlike the discovery-injection array at the + // other call site, which holds only newly discovered tools and made an earlier + // version of this gate always false. + const delegationAvailable = tools.some((t) => t.name === "sessions_spawn"); return tools.map((tool) => ({ ...tool, async execute( @@ -200,7 +205,9 @@ export function createJitGuideWrapper( _ctx: ExtensionContext, ): Promise> { const result = await tool.execute(toolCallId, params, signal, onUpdate, _ctx); - return wrapToolResultWithGuide(tool.name, result, deliveredGuides, logger); + return wrapToolResultWithGuide(tool.name, result, deliveredGuides, logger, { + delegationAvailable, + }); }, })); } From 6e74d57b4d60ba600faca802bf2b67202436969f Mon Sep 17 00:00:00 2001 From: comis-agent Date: Mon, 27 Jul 2026 23:04:20 +0300 Subject: [PATCH 28/33] fix(agent): tell the delegation policy about the child's tool profile MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A spawned sub-agent gets a RESTRICTED default profile: the parent's MCP tools and `message` are outside it. The Task Delegation policy told the model to spawn and said nothing about tool_groups, so the first spawn of any real task failed with Required tools unreachable: mcp__…, message. … outside this sub-agent's profile. Re-spawn with tool_groups:['full']. before doing any work, degrading the turn on the way to recovering. Observed on the first delegation this runtime ever performed — the failure was invisible until the preceding fix made delegation reachable at all. State it in "How to Delegate", where the model reads the rest of the spawn recipe: which two surfaces sit outside the child profile, and what to pass when the child needs them. --- .../sections/tool-descriptions.test.ts | 22 +++++++++++++++++++ .../bootstrap/sections/tool-descriptions.ts | 15 ++++++++----- 2 files changed, 31 insertions(+), 6 deletions(-) diff --git a/packages/agent/src/bootstrap/sections/tool-descriptions.test.ts b/packages/agent/src/bootstrap/sections/tool-descriptions.test.ts index 3b498821c1..6c3bdc334d 100644 --- a/packages/agent/src/bootstrap/sections/tool-descriptions.test.ts +++ b/packages/agent/src/bootstrap/sections/tool-descriptions.test.ts @@ -627,3 +627,25 @@ describe("SYSTEM_PROMPT_GUIDES trigger reachability", () => { expect(lean as string).toMatch(/parallel|multiple/i); }); }); + +/** + * A spawned sub-agent gets a RESTRICTED default profile: the parent's MCP tools + * and `message` are outside it. The delegation policy told the model to spawn but + * never mentioned tool_groups, so the first spawn of any real task fails with + * "Required tools unreachable … Re-spawn with tool_groups:['full']" and the turn + * degrades before recovering. + * + * Observed live on the first delegation the runtime ever performed. + */ +describe("Task Delegation policy covers the child tool profile", () => { + const guide = SYSTEM_PROMPT_GUIDES.sessions_spawn!; + + it("tells the model to pass tool_groups when the child needs them", () => { + expect(guide).toContain("tool_groups"); + }); + + it("names the two surfaces that are outside the default child profile", () => { + expect(guide).toMatch(/MCP/); + expect(guide).toMatch(/message/); + }); +}); diff --git a/packages/agent/src/bootstrap/sections/tool-descriptions.ts b/packages/agent/src/bootstrap/sections/tool-descriptions.ts index 8228ed19e6..abb1722a88 100644 --- a/packages/agent/src/bootstrap/sections/tool-descriptions.ts +++ b/packages/agent/src/bootstrap/sections/tool-descriptions.ts @@ -631,12 +631,15 @@ You MUST delegate tasks to a sub-agent when the work matches ANY of these criter ### How to Delegate 1. Use \`sessions_spawn\` with a **goal-oriented** task description; every spawn runs in the background -2. Describe WHAT to accomplish, not HOW -- the sub-agent has its own skills and will read SKILL.md itself -3. Do NOT copy-paste skill instructions, shell commands, or step-by-step procedures into the task -4. Include user context the sub-agent needs (e.g., desired style, dimensions, topic) but not tool instructions -5. Result delivery is bound automatically to the authenticated request route; do not supply route identifiers -6. Tell the user the task is delegated and give them the runId -7. Continue the conversation -- the result will be announced automatically when done +2. A sub-agent gets a RESTRICTED default profile: MCP tools and \`message\` are OUTSIDE it. When the child + must call an MCP tool or deliver the result itself, pass \`tool_groups: ['full']\` on the spawn -- + otherwise it fails with "Required tools unreachable" before doing any work +3. Describe WHAT to accomplish, not HOW -- the sub-agent has its own skills and will read SKILL.md itself +4. Do NOT copy-paste skill instructions, shell commands, or step-by-step procedures into the task +5. Include user context the sub-agent needs (e.g., desired style, dimensions, topic) but not tool instructions +6. Result delivery is bound automatically to the authenticated request route; do not supply route identifiers +7. Tell the user the task is delegated and give them the runId +8. Continue the conversation -- the result will be announced automatically when done ### Parallel Sub-Agents When a task has independent subtasks, spawn multiple sub-agents in parallel: From 83448bbde83df71f0b5513567de9271a251cf67d Mon Sep 17 00:00:00 2001 From: comis-agent Date: Mon, 27 Jul 2026 23:49:08 +0300 Subject: [PATCH 29/33] fix(agent): put the delegation directive in the engine prompt, before the decision MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The delegation policy rode a TOOL RESULT, so it landed AFTER the model had read the request, chosen an approach and issued its first call. Measured on a real Telegram turn with Opus 4.8: the policy arrived on the first MCP tool result while the model was already working inline, and it simply carried on — the user got no "running, I'll send it when ready" and no window to keep interacting. That also explains why the same code behaved differently per model. gpt-5.6-sol happened to re-plan after the first tool result and spawned; Opus committed to its plan and did not. A directive that must shape the FIRST decision cannot be delivered as a consequence of that decision. Move it into the engine kernel, gated on sessions_spawn being on the surface — telling an agent without the tool to delegate is worse than silence. Sourced from the CACHE-STABLE tool snapshot, not the live list, so the cached system prefix does not flip between turns when MCP servers connect or disconnect. A universal orchestration mechanism with no domain assumptions, which is what the engine prompt is allowed to carry. The JIT guide stays as the detailed recipe. --- .../src/bootstrap/system-prompt-assembler.ts | 3 ++ .../src/executor/prompt-assembly-runtime.ts | 4 +++ .../src/executor/prompt-compiler.test.ts | 36 +++++++++++++++++++ .../agent/src/executor/prompt-compiler.ts | 23 ++++++++++-- 4 files changed, 64 insertions(+), 2 deletions(-) diff --git a/packages/agent/src/bootstrap/system-prompt-assembler.ts b/packages/agent/src/bootstrap/system-prompt-assembler.ts index 94ef5e65c0..aa0b91fa32 100644 --- a/packages/agent/src/bootstrap/system-prompt-assembler.ts +++ b/packages/agent/src/bootstrap/system-prompt-assembler.ts @@ -34,6 +34,8 @@ export interface AssemblerParams { promptSkillsXml?: string; activePromptSkillContent?: string; reasoningTagHint?: boolean; + /** True when sessions_spawn is on the surface — see PromptCompilerInput. */ + delegationAvailable?: boolean; } function sha256Hex(value: string): string { @@ -92,6 +94,7 @@ function runtimeSections(params: AssemblerParams): readonly RuntimePromptSection export function compileRichSystemPrompt(params: AssemblerParams): CompiledExecutionPrompt { return compileExecutionPrompt({ mode: params.promptMode ?? "full", + ...(params.delegationAvailable === undefined ? {} : { delegationAvailable: params.delegationAvailable }), operatorPolicy: params.instructionSections ?? legacyBootstrapSections(params.bootstrapFiles ?? []), runtimeSections: runtimeSections(params), diff --git a/packages/agent/src/executor/prompt-assembly-runtime.ts b/packages/agent/src/executor/prompt-assembly-runtime.ts index 4dd4749573..3e9df80323 100644 --- a/packages/agent/src/executor/prompt-assembly-runtime.ts +++ b/packages/agent/src/executor/prompt-assembly-runtime.ts @@ -692,6 +692,10 @@ export async function assembleExecutionPrompt(params: PromptAssemblyParams): Pro // One typed compiler input feeds the monolithic and cache-block views. const assemblerParams: import("../bootstrap/index.js").AssemblerParams = { promptMode, + // From the CACHE-STABLE tool snapshot, not the live list: the directive is + // part of the cached system prefix, so it must not flip between turns when + // MCP servers connect or disconnect. + delegationAvailable: stableToolNames.includes("sessions_spawn"), instructionSections: deps.workspacePolicySnapshot.sections, bootstrapFiles: bootstrapContextFiles, promptSkillsXml, // skills XML in semiStableBody for 1h cache diff --git a/packages/agent/src/executor/prompt-compiler.test.ts b/packages/agent/src/executor/prompt-compiler.test.ts index 8d6c3d32fb..e07440f79c 100644 --- a/packages/agent/src/executor/prompt-compiler.test.ts +++ b/packages/agent/src/executor/prompt-compiler.test.ts @@ -105,3 +105,39 @@ describe("compileExecutionPrompt", () => { .toBe("included"); }); }); + +/** + * The delegation policy previously rode a TOOL RESULT, which lands AFTER the model + * has read the request, chosen an approach and issued its first call. Measured on + * a real Telegram turn: the policy arrived on the first MCP tool result while the + * model was already working inline, and it carried on inline to completion — the + * user got no "running, I'll send it when ready" and no chance to interact. + * + * A directive that must shape the FIRST decision belongs in the system prompt. + */ +describe("compileExecutionPrompt delegation directive", () => { + + + it("puts the delegation directive in the engine kernel when spawning is available", () => { + const out = compileExecutionPrompt(makeInput({ delegationAvailable: true })); + expect(out.stableEnginePrefix).toContain("sessions_spawn"); + expect(out.stableEnginePrefix).toMatch(/30 ?seconds|~30/); + }); + + it("tells the model to end the turn rather than work inline", () => { + const out = compileExecutionPrompt(makeInput({ delegationAvailable: true })); + expect(out.stableEnginePrefix).toMatch(/end the turn/i); + expect(out.stableEnginePrefix).toMatch(/inline/i); + }); + + it("stays silent for an agent without the tool", () => { + const out = compileExecutionPrompt(makeInput({ delegationAvailable: false })); + expect(out.stableEnginePrefix).not.toContain("sessions_spawn"); + }); + + it("adds nothing when the flag is omitted", () => { + const out = compileExecutionPrompt(makeInput()).stableEnginePrefix; + expect(out.length).toBeGreaterThan(0); + expect(out).not.toContain("sessions_spawn"); + }); +}); diff --git a/packages/agent/src/executor/prompt-compiler.ts b/packages/agent/src/executor/prompt-compiler.ts index 9dc45428d8..45022b1371 100644 --- a/packages/agent/src/executor/prompt-compiler.ts +++ b/packages/agent/src/executor/prompt-compiler.ts @@ -36,6 +36,19 @@ export interface PromptCompilerInput { readonly operatorPolicy: readonly InstructionSection[]; readonly runtimeSections: readonly RuntimePromptSection[]; readonly requireFinalTags?: boolean; + /** + * True when `sessions_spawn` is on the agent's surface. + * + * The delegation policy used to ride a TOOL RESULT, which lands AFTER the model + * has already read the request, chosen an approach and issued its first call — + * measured live: the policy arrived on the first MCP result while the model was + * already working inline, and it simply carried on. A directive that must shape + * the FIRST decision has to be in the system prompt, before any of it happens. + * + * Gated because telling an agent without the tool to delegate is worse than + * silence. Universal orchestration mechanism, no domain assumptions. + */ + readonly delegationAvailable?: boolean; } export interface PromptCompileSectionReport { @@ -110,9 +123,15 @@ function joinSections(sections: readonly string[]): string { } export function compileExecutionPrompt(input: PromptCompilerInput): CompiledExecutionPrompt { - const engineContent = input.requireFinalTags + const delegationDirective = input.delegationAvailable === true + ? "\n- Work that needs more than ~30 seconds of tool time, generates media, writes 3+ files," + + " or runs 4+ dependent steps: call `sessions_spawn` FIRST, tell the user it is running and" + + " that you will send the result when ready, then end the turn. Do not grind it out inline." + + " Independent parts spawn in parallel in one response." + : ""; + const engineContent = (input.requireFinalTags ? `${ENGINE_KERNEL}\n- Put user-visible output inside the provider's required final-output tags.` - : ENGINE_KERNEL; + : ENGINE_KERNEL) + delegationDirective; const engineHash = sha256Hex(engineContent); const reports: PromptCompileSectionReport[] = [{ id: "engine:kernel", From aee8c6aef6197493c95f0b0db7c474fbd258aee6 Mon Sep 17 00:00:00 2001 From: comis-agent Date: Tue, 28 Jul 2026 00:11:06 +0300 Subject: [PATCH 30/33] fix(agent): tell the message tool not to double-post the turn's final text MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The turn's final assistant text is delivered to the channel automatically. When the agent ALSO delivers user-facing content with the `message` tool, both go out. Observed on real Telegram: a file arrived with its caption, immediately followed by a second message restating it. The trajectory showed message x2 against THREE dispatched deliveries — one extra per exchange. The runtime already honours a silent sentinel (isSilentResponse) for exactly this case; the model was simply never told to use it after sending. State the contract where it decides: deliver via `message`, then reply NO_REPLY. Prompt-level, so it carries the same caveat the delegation directive just proved — a model may not comply. The durable fix is for the runtime to suppress the final delivery when the agent already delivered to the same channel this turn, which needs the message tool's target channel threaded to post-execution to avoid suppressing a legitimate reply after a cross-channel send. --- .../sections/tool-descriptions.test.ts | 29 +++++++++++++++++++ .../bootstrap/sections/tool-descriptions.ts | 5 +++- 2 files changed, 33 insertions(+), 1 deletion(-) diff --git a/packages/agent/src/bootstrap/sections/tool-descriptions.test.ts b/packages/agent/src/bootstrap/sections/tool-descriptions.test.ts index 6c3bdc334d..c75f52c518 100644 --- a/packages/agent/src/bootstrap/sections/tool-descriptions.test.ts +++ b/packages/agent/src/bootstrap/sections/tool-descriptions.test.ts @@ -649,3 +649,32 @@ describe("Task Delegation policy covers the child tool profile", () => { expect(guide).toMatch(/message/); }); }); + +/** + * The turn's final assistant text is delivered to the channel automatically. When + * the agent ALSO delivers user-facing content with the `message` tool, both go + * out and the user sees the same thing twice. + * + * Observed live on real Telegram: a file arrived with its caption, immediately + * followed by a second message restating it ("הקובץ נשלח ✅ …"). The trajectory + * showed message ×2 but THREE dispatched deliveries. + * + * The runtime already honours a silent sentinel (isSilentResponse) — the model + * was simply never told to use it after sending. + */ +describe("message tool states the no-double-post contract", () => { + const ctx: ToolDescriptionContext = { modelTier: "large", channelType: "telegram" }; + const desc = resolveDescription({ name: "message" }, LEAN_TOOL_DESCRIPTIONS, ctx); + + it("tells the model to return the silent sentinel after delivering", () => { + expect(desc).toContain("NO_REPLY"); + }); + + it("says why — the final text is sent as well", () => { + expect(desc).toMatch(/also|too|double|twice/i); + }); + + it("stays within the lean description budget", () => { + expect(desc.length).toBeLessThanOrEqual(300); + }); +}); diff --git a/packages/agent/src/bootstrap/sections/tool-descriptions.ts b/packages/agent/src/bootstrap/sections/tool-descriptions.ts index abb1722a88..b02adfcc5e 100644 --- a/packages/agent/src/bootstrap/sections/tool-descriptions.ts +++ b/packages/agent/src/bootstrap/sections/tool-descriptions.ts @@ -141,7 +141,10 @@ export const LEAN_TOOL_DESCRIPTIONS: Record { const ch = ctx.channelType ?? "chat"; - return `Send, reply, react, edit, delete, fetch messages on ${ch}. For inter-session messaging, use sessions_send.`; + // The turn's final text is delivered automatically; an agent that also sends + // here double-posts. The runtime honours the silent sentinel — say so. + return `Send, reply, react, edit, delete, fetch messages on ${ch}. For inter-session messaging, use sessions_send.` + + ` After delivering user-facing content here, reply NO_REPLY: the turn's final text is sent too, so restating it double-posts.`; }, // ----- Sessions ----- From e40e34b2a4567e35c916565900c903d5dc052f4c Mon Sep 17 00:00:00 2001 From: comis-agent Date: Tue, 28 Jul 2026 01:23:47 +0300 Subject: [PATCH 31/33] fix: preserve truthful incident recovery outcomes Combine regression tests and production changes in one review commit because the round includes secret-handling corrections. --- docs/reference/json-rpc.mdx | 2 +- .../background-task-manager.test.ts | 26 ++++++- .../src/background/background-task-manager.ts | 37 ++++++---- .../background/background-task-persistence.ts | 4 ++ .../src/background/background-task-types.ts | 15 +++- .../src/background/completion-runner.test.ts | 5 +- .../src/context-engine/lcd-assembler.test.ts | 32 +++++++++ .../agent/src/context-engine/lcd-assembler.ts | 27 +++++-- .../lcd-fresh-tail-bound.test.ts | 21 +++++- .../context-engine/lcd-fresh-tail-bound.ts | 49 +++++++++++-- .../src/context-engine/lcd-preflight.test.ts | 20 ++++++ .../agent/src/context-engine/lcd-preflight.ts | 17 +++-- .../executor/executor-response-filter.test.ts | 34 +++++++-- .../src/executor/executor-response-filter.ts | 12 ++++ .../src/executor/pi-executor/pi-executor.ts | 1 + .../background-failure-attribution.test.ts | 16 +++++ .../safety/background-failure-attribution.ts | 10 ++- packages/core/src/activity/activity-event.ts | 2 +- packages/core/src/activity/turn-outcome.ts | 14 ++-- .../api-contracts/incident-report-sections.ts | 10 ++- packages/core/src/event-bus/events-infra.ts | 7 +- .../core/src/event-bus/events-messaging.ts | 20 ++---- .../src/security/secret-detection.test.ts | 36 ++++------ .../core/src/security/secret-detection.ts | 63 +---------------- .../obs-explain-fresh-tail-verdict.test.ts | 42 ++++++----- .../obs-explain-fresh-tail-verdict.ts | 56 ++++----------- .../obs-handlers/obs-explain-heuristics.ts | 11 ++- packages/daemon/src/daemon.ts | 11 +-- .../src/wiring/proactive-degrade.test.ts | 50 +++++++++++++ .../daemon/src/wiring/proactive-degrade.ts | 70 ++++++++++++++----- .../setup-capability-endpoint-boot.test.ts | 3 + .../wiring/setup-capability-endpoint-boot.ts | 43 ++++++++---- .../src/activity/activity-stream.test.ts | 7 +- .../src/activity/activity-stream.ts | 7 +- .../src/trajectory/event-bus-bridge.test.ts | 8 +++ .../src/trajectory/translate-payload.test.ts | 3 +- .../src/trajectory/translate-payload.ts | 13 +++- .../activity-turn-coordinator.test.ts | 25 ++++++- .../execution/activity-turn-coordinator.ts | 24 ++++--- .../src/execution/execution-pipeline.ts | 7 +- .../platform-tools/tools/gateway-tool.test.ts | 30 ++++++++ .../src/platform-tools/tools/gateway-tool.ts | 18 +++-- .../src/skills/bridge/mcp-tool-bridge.test.ts | 40 ++++++----- .../src/skills/bridge/mcp-tool-bridge.ts | 20 +++++- .../tools/builtin/exec-diagnostics.test.ts | 5 +- .../src/tools/builtin/exec-diagnostics.ts | 14 +--- ...mon-boot-degrades-without-autonomy.test.ts | 4 +- 47 files changed, 669 insertions(+), 322 deletions(-) create mode 100644 packages/daemon/src/wiring/proactive-degrade.test.ts diff --git a/docs/reference/json-rpc.mdx b/docs/reference/json-rpc.mdx index af8ce7c9b5..b0c4970da6 100644 --- a/docs/reference/json-rpc.mdx +++ b/docs/reference/json-rpc.mdx @@ -784,7 +784,7 @@ durably reset a task transition, the higher-priority reconciliation authority and the `health_signal:background_task_recovery_failed` system-health signal. -For `context_exhausted` sessions the report carries an optional **`contextBudget`** section -- the terminal per-LLM-call budget equation extracted from the trajectory's `context.budget` records (last record wins): `{ windowTokens, rawContextWindowTokens, windowCapSource, systemTokens, freshTailTokens, budgetedHistoryTokens, keptCount, assembledInputTokens, outputHeadroom, verdict, freshTailSteps?, freshTailStepsConfigured? }`. `freshTailSteps` / `freshTailStepsConfigured` are the **effective vs operator-configured** count of trailing conversation steps kept verbatim (`contextEngine.freshTailTurns`). The verbatim tail is bounded by **step count, not tokens**, but that count is a floor: the tail also always reaches back to the store's persisted horizon, so the current turn's own request cannot fall out of context however many tool round-trips the turn runs. When the effective value is below the configured one, `likelyRootCause` stamps a `fresh_tail_clamped` verdict naming both numbers and the knob. `windowCapSource` names what clamped the effective window below the model's configured `contextWindow`: the exact `contextEngine.budget.*` knob (`effectiveContextCapSmall` / `effectiveContextCapNano`), `served` (the Ollama-served `num_ctx` bound the window -- fix with `OLLAMA_CONTEXT_LENGTH= ollama serve` or a Modelfile `PARAMETER num_ctx`), `capabilityClass` (the operator's `providers.entries..capabilities.capabilityClass` pin bound the window -- pin a higher class or remove the pin; the `contextEngine.budget.*` caps do not move this bind), or `none`. `verdict` is the fit-check outcome (`fits` / `downshifted` / `exhausted`). When present, `likelyRootCause` is numbers-backed: it states the assembled-vs-window totals, the binding cap, the system+tools share of the window, and the kept-history count, and its suggested steps name the binding lever (the budget knob to raise, the Ollama serve knobs, or the `capabilityClass` pin) or recommend reducing the active tool surface when tool schemas dominate. The section is absent when the trajectory carries no `context.budget` records; the verdict then falls back to the generic wording. +For `context_exhausted` sessions the report carries an optional **`contextBudget`** section -- the terminal per-LLM-call budget equation extracted from the trajectory's `context.budget` records (last record wins): `{ windowTokens, rawContextWindowTokens, windowCapSource, systemTokens, freshTailTokens, budgetedHistoryTokens, keptCount, assembledInputTokens, outputHeadroom, verdict, freshTailSteps?, freshTailStepsConfigured?, originatingRequestRetained?, freshTailTrimmedCount? }`. `freshTailSteps` / `freshTailStepsConfigured` are the **effective vs operator-configured** count of trailing conversation steps kept verbatim (`contextEngine.freshTailTurns`). The verbatim tail also reaches back to the store's persisted horizon, and residual trimming pins the current turn's originating request while discarding completed tool segments. `originatingRequestRetained` records the resulting coverage directly; only an explicit `false` produces the `fresh_tail_origin_lost` root-cause verdict, while a step-count clamp alone is not treated as evidence of loss. `windowCapSource` names what clamped the effective window below the model's configured `contextWindow`: the exact `contextEngine.budget.*` knob (`effectiveContextCapSmall` / `effectiveContextCapNano`), `served` (the Ollama-served `num_ctx` bound the window -- fix with `OLLAMA_CONTEXT_LENGTH= ollama serve` or a Modelfile `PARAMETER num_ctx`), `capabilityClass` (the operator's `providers.entries..capabilities.capabilityClass` pin bound the window -- pin a higher class or remove the pin; the `contextEngine.budget.*` caps do not move this bind), or `none`. `verdict` is the fit-check outcome (`fits` / `downshifted` / `exhausted`). When present, `likelyRootCause` is numbers-backed: it states the assembled-vs-window totals, the binding cap, the system+tools share of the window, and the kept-history count, and its suggested steps name the binding lever (the budget knob to raise, the Ollama serve knobs, or the `capabilityClass` pin) or recommend reducing the active tool surface when tool schemas dominate. The section is absent when the trajectory carries no `context.budget` records; the verdict then falls back to the generic wording. When the session ran any memory recalls the report carries an optional **`recall`** section -- the memory-recall outcome aggregated over the trajectory's `memory.recalled` records: `{ recalls, zeroHits, lastLanes, lastFinalCount, rerankerAvailable, crossUserRecalls?, lastCrossUserCount? }` (counts and booleans only -- never the query text or any recalled memory body). `recalls` is how many recalls ran, `zeroHits` how many returned **no** injected memories (a recall miss), and the `last*` fields describe the terminal recall (its lane count and final injected-memory count). **`crossUserRecalls`** (present only when `> 0`) is how many recalls injected at least one memory scoped to a **different user** than the conversation -- the cross-sender privacy signal for agent-scoped recall (one shared agent surfacing sender A's memory into sender B's turn), so "was cross-sender data injected into this turn's context?" is answerable from the always-on report without pre-enabling the opt-in `diagnostics.recallTrace` artifact; **`lastCrossUserCount`** is the terminal recall's cross-user injected count. Counts only -- never the user-ids or bodies. This drives a dedicated **`recall_miss`** `likelyRootCause` verdict: a degraded session whose recalls **all** missed (`zeroHits === recalls`) and that matched no tool/context/breaker cause is root-caused as a recall miss, pointing the operator at the recall **scope** (agent- vs user-scoped) and, for non-Latin queries, the trigram-twin lanes (see [`comis system-health`](#obs-system-health) `health_signal` / `config_posture` embedder). A zero-hit recall on a **healthy** turn is benign -- the agent simply did not need memory -- and never names a cause. The section is absent for sessions with no recall records. diff --git a/packages/agent/src/background/background-task-manager.test.ts b/packages/agent/src/background/background-task-manager.test.ts index 2930b3b42f..30d7c56c7a 100644 --- a/packages/agent/src/background/background-task-manager.test.ts +++ b/packages/agent/src/background/background-task-manager.test.ts @@ -370,11 +370,13 @@ describe("BackgroundTaskManager", () => { const r = manager.promote("report", new Promise(() => {}), new AbortController(), origin, undefined, CORR); expect(r.ok).toBe(true); if (!r.ok) return; - manager.fail(r.value, new Error("timed out")); + expect(loadTask(dataDir, "agent-1", r.value)).toMatchObject(CORR); + manager.fail(r.value, new Error("timed out"), "timeout"); expect(eventBus.emit).toHaveBeenCalledWith( "background_task:failed", - expect.objectContaining(CORR), + expect.objectContaining({ ...CORR, errorKind: "timeout" }), ); + expect(loadTask(dataDir, "agent-1", r.value)?.errorKind).toBe("timeout"); }); it("complete() emits the same correlation", () => { @@ -701,6 +703,7 @@ describe("BackgroundTaskManager", () => { const task = manager.getTask(result.value); expect(task!.status).toBe("failed"); expect(task!.error).toContain("Hard timeout exceeded"); + expect(task!.errorKind).toBe("timeout"); expect(ac.signal.aborted).toBe(true); } finally { vi.useRealTimers(); @@ -760,6 +763,9 @@ describe("BackgroundTaskManager", () => { origin: buildOrigin({ agentId: "a1" }), continuationExecutionId: "recovered-1", dispatchAttempts: 0, + toolCallId: "call-recovered-1", + sessionKey: "session-recovered-1", + traceId: "trace-recovered-1", }; persistTaskSync(dataDir, task); @@ -779,12 +785,28 @@ describe("BackgroundTaskManager", () => { expect(recovered).toBeDefined(); expect(recovered!.status).toBe("failed"); expect(recovered!.error).toBe("Daemon restarted while task was running"); + expect(recovered).toMatchObject({ + toolCallId: "call-recovered-1", + sessionKey: "session-recovered-1", + traceId: "trace-recovered-1", + errorKind: "internal", + }); + expect(loadTask(dataDir, "a1", "recovered-1")).toMatchObject({ + toolCallId: "call-recovered-1", + sessionKey: "session-recovered-1", + traceId: "trace-recovered-1", + errorKind: "internal", + }); expect((eventBus.emit as ReturnType)).toHaveBeenCalledWith( "background_task:failed", expect.objectContaining({ taskId: "recovered-1", error: "Daemon restarted while task was running", + toolCallId: "call-recovered-1", + sessionKey: "session-recovered-1", + traceId: "trace-recovered-1", + errorKind: "internal", }), ); diff --git a/packages/agent/src/background/background-task-manager.ts b/packages/agent/src/background/background-task-manager.ts index b7d4adc3c4..414b7ae5b9 100644 --- a/packages/agent/src/background/background-task-manager.ts +++ b/packages/agent/src/background/background-task-manager.ts @@ -11,6 +11,7 @@ import { type ClockPort, type TimerHandle, type TimerPort, + type ErrorKind, } from "@comis/core"; import { persistTaskAtomically, @@ -76,7 +77,7 @@ export interface BackgroundTaskManager { correlation?: { toolCallId?: string; sessionKey?: string; traceId?: string }, ): Result; complete(taskId: string, result: unknown): Result; - fail(taskId: string, error: unknown): Result; + fail(taskId: string, error: unknown, errorKind?: ErrorKind): Result; cancel(taskId: string): Result; getTask(taskId: string): BackgroundTask | undefined; waitForTask( @@ -251,6 +252,7 @@ export function createBackgroundTaskManager(opts: BackgroundTaskManagerOpts): Ba completedAt: terminal.completedAt, ...(terminal.result === undefined ? {} : { result: terminal.result }), ...(terminal.error === undefined ? {} : { error: terminal.error }), + ...(terminal.errorKind === undefined ? {} : { errorKind: terminal.errorKind }), }; const persisted = persistTaskAtomically(dataDir, candidate, persistenceOps); if (!persisted.ok) { @@ -271,6 +273,7 @@ export function createBackgroundTaskManager(opts: BackgroundTaskManagerOpts): Ba task.completedAt = terminal.completedAt; if (terminal.result !== undefined) task.result = terminal.result; if (terminal.error !== undefined) task.error = terminal.error; + if (terminal.errorKind !== undefined) task.errorKind = terminal.errorKind; task._pendingTerminal = undefined; terminalRetryTimers.get(task.id)?.cancel(); terminalRetryTimers.delete(task.id); @@ -371,7 +374,7 @@ export function createBackgroundTaskManager(opts: BackgroundTaskManagerOpts): Ba const timer = timers.setTimeout(() => { if (task.status === "running") { ac.abort(); - manager.fail(taskId, new Error("Hard timeout exceeded")); + manager.fail(taskId, new Error("Hard timeout exceeded"), "timeout"); } }, resolveMaxBackgroundDurationMs(agentId)); timer.unref(); @@ -420,7 +423,7 @@ export function createBackgroundTaskManager(opts: BackgroundTaskManagerOpts): Ba return ok(undefined); }, - fail(taskId, error) { + fail(taskId, error, errorKind = "dependency") { const task = tasks.get(taskId); if (!task || task.status !== "running") return ok(undefined); if (task._pendingTerminal !== undefined && task._pendingTerminal.status !== "failed") { @@ -432,6 +435,7 @@ export function createBackgroundTaskManager(opts: BackgroundTaskManagerOpts): Ba status: "failed" as const, completedAt: clock.now(), error: error instanceof Error ? error.message : String(error), + errorKind, }; const committed = commitTerminal(task, terminal); if (!committed.ok) return committed; @@ -441,6 +445,7 @@ export function createBackgroundTaskManager(opts: BackgroundTaskManagerOpts): Ba taskId, toolName: task.toolName, error: terminal.error ?? "Background task failed", + errorKind: terminal.errorKind ?? errorKind, durationMs, origin: task.origin, timestamp: clock.now(), @@ -556,6 +561,7 @@ export function createBackgroundTaskManager(opts: BackgroundTaskManagerOpts): Ba ...persisted, status: "failed" as const, error: "Daemon restarted while task was running", + errorKind: "internal" as const, completedAt: clock.now(), }; const committed = persistTaskAtomically(dataDir, terminal, persistenceOps); @@ -564,6 +570,7 @@ export function createBackgroundTaskManager(opts: BackgroundTaskManagerOpts): Ba status: "failed", completedAt: terminal.completedAt, error: terminal.error, + errorKind: terminal.errorKind, }; tasks.set(task.id, task); scheduleTerminalRetry(task.id); @@ -590,32 +597,33 @@ export function createBackgroundTaskManager(opts: BackgroundTaskManagerOpts): Ba dispatchPreserved++; continue; } - if (persisted.status === "completed") { + if (task.status === "completed") { count++; emitObservationalEventSafely({ eventBus, logger }, "background_task:completed", { agentId: task.origin.turnScope.conversation.agentId, taskId: task.id, toolName: task.toolName, - durationMs: (persisted.completedAt ?? clock.now()) - persisted.startedAt, + durationMs: (task.completedAt ?? clock.now()) - task.startedAt, origin: task.origin, timestamp: clock.now(), - ...(task.toolCallId !== undefined ? { toolCallId: task.toolCallId } : {}), - ...(task.sessionKey !== undefined ? { sessionKey: task.sessionKey } : {}), - ...(task.traceId !== undefined ? { traceId: task.traceId } : {}), + ...(task.toolCallId !== undefined ? { toolCallId: task.toolCallId } : {}), + ...(task.sessionKey !== undefined ? { sessionKey: task.sessionKey } : {}), + ...(task.traceId !== undefined ? { traceId: task.traceId } : {}), }); - } else if (persisted.status === "failed") { + } else if (task.status === "failed") { count++; emitObservationalEventSafely({ eventBus, logger }, "background_task:failed", { agentId: task.origin.turnScope.conversation.agentId, taskId: task.id, toolName: task.toolName, - error: persisted.error ?? "Background task failed", - durationMs: (persisted.completedAt ?? clock.now()) - persisted.startedAt, + error: task.error ?? "Background task failed", + errorKind: task.errorKind ?? "dependency", + durationMs: (task.completedAt ?? clock.now()) - task.startedAt, origin: task.origin, timestamp: clock.now(), - ...(task.toolCallId !== undefined ? { toolCallId: task.toolCallId } : {}), - ...(task.sessionKey !== undefined ? { sessionKey: task.sessionKey } : {}), - ...(task.traceId !== undefined ? { traceId: task.traceId } : {}), + ...(task.toolCallId !== undefined ? { toolCallId: task.toolCallId } : {}), + ...(task.sessionKey !== undefined ? { sessionKey: task.sessionKey } : {}), + ...(task.traceId !== undefined ? { traceId: task.traceId } : {}), }); } } @@ -811,6 +819,7 @@ export function createBackgroundTaskManager(opts: BackgroundTaskManagerOpts): Ba emitObservationalEventSafely({ eventBus, logger }, event, { ...common, error: current.error ?? "Background task failed", + errorKind: current.errorKind ?? "dependency", }); } }, delayMs); diff --git a/packages/agent/src/background/background-task-persistence.ts b/packages/agent/src/background/background-task-persistence.ts index 9c84fcfd49..77ceefe8e1 100644 --- a/packages/agent/src/background/background-task-persistence.ts +++ b/packages/agent/src/background/background-task-persistence.ts @@ -61,6 +61,10 @@ export function toPersistedState(task: BackgroundTask | PersistedTaskState): Per ...(task.finalizedResult !== undefined && { finalizedResult: task.finalizedResult }), ...(task.notificationPolicy !== undefined && { notificationPolicy: task.notificationPolicy }), ...(task.dispatchState !== undefined && { dispatchState: task.dispatchState }), + ...(task.toolCallId !== undefined && { toolCallId: task.toolCallId }), + ...(task.sessionKey !== undefined && { sessionKey: task.sessionKey }), + ...(task.traceId !== undefined && { traceId: task.traceId }), + ...(task.errorKind !== undefined && { errorKind: task.errorKind }), }; } diff --git a/packages/agent/src/background/background-task-types.ts b/packages/agent/src/background/background-task-types.ts index 179cf69baa..550a61801c 100644 --- a/packages/agent/src/background/background-task-types.ts +++ b/packages/agent/src/background/background-task-types.ts @@ -134,6 +134,13 @@ export const PersistedTaskStateSchema = z.strictObject({ dispatchAttempts: z.number().int().nonnegative(), continuationOutbox: BackgroundContinuationOutboxSchema.optional(), finalizedResult: BackgroundFinalizedResultSchema.optional(), + toolCallId: z.string().min(1).max(512).optional(), + sessionKey: z.string().min(1).max(2048).optional(), + traceId: z.string().min(1).max(512).optional(), + errorKind: z.enum([ + "config", "network", "auth", "validation", "precondition", "timeout", + "resource", "dependency", "internal", "platform", "sandbox_unavailable", + ]).optional(), }); // @optional-field-count: Lifecycle fields are conditional by task state; underscored fields exist only while execution or terminal persistence is in flight. @@ -155,10 +162,11 @@ export interface BackgroundTask { /** Correlation to the originating tool call + turn, captured at promote time * and PERSISTED (ids only — no content), so the terminal * `background_task:{completed,failed}` event can close the right activity - * card even across a daemon restart. Absent on pre-upgrade records. */ + * card even across a daemon restart. */ toolCallId?: string; sessionKey?: string; traceId?: string; + errorKind?: ErrorKind; /** Durable completion lifecycle. */ dispatchState?: BackgroundSessionState; continuationExecutionId: string; @@ -176,6 +184,7 @@ export interface BackgroundTask { completedAt: number; result?: string; error?: string; + errorKind?: ErrorKind; }; _ownsCounterSlot?: boolean; } @@ -205,6 +214,10 @@ export interface PersistedTaskState { dispatchAttempts: number; continuationOutbox?: BackgroundContinuationOutbox; finalizedResult?: BackgroundFinalizedResult; + toolCallId?: string; + sessionKey?: string; + traceId?: string; + errorKind?: ErrorKind; } export function isClosedBackgroundTask(task: Pick): boolean { diff --git a/packages/agent/src/background/completion-runner.test.ts b/packages/agent/src/background/completion-runner.test.ts index 1272a604c0..54d6d82fa2 100644 --- a/packages/agent/src/background/completion-runner.test.ts +++ b/packages/agent/src/background/completion-runner.test.ts @@ -683,6 +683,7 @@ describe("createBackgroundCompletionRunner", () => { taskId: task.id, toolName: task.toolName, error: task.error!, + errorKind: "dependency", durationMs: 1, origin: task.origin, timestamp: 3, @@ -716,7 +717,7 @@ describe("createBackgroundCompletionRunner", () => { const runner = build(); eventBus.emit("background_task:failed", { agentId: task.origin.turnScope.conversation.agentId, taskId: task.id, toolName: task.toolName, - error: "boom", durationMs: 1, origin: task.origin, timestamp: 3, + error: "boom", errorKind: "dependency", durationMs: 1, origin: task.origin, timestamp: 3, }); await new Promise((r) => setImmediate(r)); await new Promise((r) => setImmediate(r)); @@ -826,7 +827,7 @@ describe("createBackgroundCompletionRunner", () => { const runner = build(); eventBus.emit("background_task:failed", { agentId: task.origin.turnScope.conversation.agentId, taskId: task.id, toolName: task.toolName, - error: task.error!, durationMs: 1, origin: task.origin, timestamp: 3, + error: task.error!, errorKind: "internal", durationMs: 1, origin: task.origin, timestamp: 3, }); await new Promise((r) => setImmediate(r)); await new Promise((r) => setImmediate(r)); diff --git a/packages/agent/src/context-engine/lcd-assembler.test.ts b/packages/agent/src/context-engine/lcd-assembler.test.ts index 057d266e28..5f95073f83 100644 --- a/packages/agent/src/context-engine/lcd-assembler.test.ts +++ b/packages/agent/src/context-engine/lcd-assembler.test.ts @@ -473,6 +473,38 @@ describe("createLcdContextEngine", () => { expect(callIds).toEqual(["tu_1", "tu_2", "tu_3"]); }); + it("residual trimming retains the in-flight request across a long tool loop", async () => { + const request = userMsg("THE-IN-FLIGHT-REQUEST") as AgentMessage; + const live = [ + request, + assistantToolCall("old-1", "read", {}) as AgentMessage, + toolResult("old-1", "read", "x".repeat(5_000)) as AgentMessage, + assistantToolCall("old-2", "read", {}) as AgentMessage, + toolResult("old-2", "read", "y".repeat(5_000)) as AgentMessage, + assistantToolCall("latest", "read", {}) as AgentMessage, + toolResult("latest", "read", "done") as AgentMessage, + ]; + const { deps } = makeDeps(store); + const windowTokens = 8_192; + deps.getModel = () => ({ reasoning: false, contextWindow: windowTokens, maxTokens: 4_096 }); + deps.getSystemTokensEstimate = () => 5_210; + deps.getThinkingLevel = () => "off"; + deps.modelProfile = { + ...FAIL_CLOSED_PROFILE, + capabilityClass: "nano", + contextWindow: windowTokens, + maxOutputTokens: 4_096, + reasoningStyle: "none", + }; + + const out = await createLcdContextEngine(dagConfig(8), deps).transformContext(live); + const serialized = JSON.stringify(out); + + expect(out).toContain(request); + expect(serialized).not.toContain('"id":"old-1"'); + expect(serialized).toContain('"id":"latest"'); + }); + it("live array SHRINKS below the store count — assembler over-includes nothing, doubles nothing", async () => { // A future heal/compaction could reassign state.messages SMALLER than the // append-only store. The assembler seam must stay robust to live.length <= diff --git a/packages/agent/src/context-engine/lcd-assembler.ts b/packages/agent/src/context-engine/lcd-assembler.ts index 59470e3ffe..08e148548e 100644 --- a/packages/agent/src/context-engine/lcd-assembler.ts +++ b/packages/agent/src/context-engine/lcd-assembler.ts @@ -431,6 +431,17 @@ export function createLcdContextEngine( const freshTailCapChars = computeFreshTailCapChars(budget.availableHistoryTokens); const bounded = boundFreshTailMessages(rawFreshTail, freshTailCapChars); let freshTail = bounded.freshTail; + const firstInFlightIndex = Math.max(0, persistedMsgCount - tailStart); + const originatingRequestIndex = freshTail.findIndex((message, index) => + index >= firstInFlightIndex + && (message as unknown as { role?: string }).role === "user", + ); + const protectedOriginIndex = originatingRequestIndex >= 0 + ? originatingRequestIndex + : undefined; + const originatingRequest = protectedOriginIndex === undefined + ? undefined + : freshTail[protectedOriginIndex]; const { boundedResults, boundedMessages, charsRemoved } = bounded; if (boundedResults > 0 || boundedMessages > 0) { // Content-free DEBUG (AGENTS.md §2.2 / the lossless-store content-free rule): @@ -480,7 +491,12 @@ export function createLcdContextEngine( logger: deps.logger, agentId: deps.agentId, sessionKey: deps.sessionKey, + protectedMessageIndex: protectedOriginIndex, }); + const originatingRequestRetained = originatingRequest === undefined + ? undefined + : freshTail.includes(originatingRequest); + const freshTailTrimmedCount = rawFreshTail.length - freshTail.length; // Eviction seam (step 4 above). Frontier/mid (relevanceFirst falsy) take // the recency eviction call — the arbiter does NOT run for them, so their @@ -596,10 +612,13 @@ export function createLcdContextEngine( windowCapSource: budget.windowCapSource, servedWindowTokens: budget.servedWindowTokens, }, - // The verbatim tail's STEP bound (effective vs the operator's configured - // value) — the number that decides whether the user's own request is - // still in context, and which was previously DEBUG-log-only. - { effective: clampedFreshTailTurns, configured: config.freshTailTurns }, + // The verbatim-tail step bound and direct origin-retention evidence. + { + effective: clampedFreshTailTurns, + configured: config.freshTailTurns, + originatingRequestRetained, + freshTailTrimmedCount, + }, ); deps.logger.info( diff --git a/packages/agent/src/context-engine/lcd-fresh-tail-bound.test.ts b/packages/agent/src/context-engine/lcd-fresh-tail-bound.test.ts index ca58effbe9..2613e5d1ae 100644 --- a/packages/agent/src/context-engine/lcd-fresh-tail-bound.test.ts +++ b/packages/agent/src/context-engine/lcd-fresh-tail-bound.test.ts @@ -8,7 +8,7 @@ */ import { describe, it, expect, vi } from "vitest"; import type { AgentMessage } from "@earendil-works/pi-agent-core"; -import { computeFreshTailCapChars, boundFreshTailMessages, boundProtectedFreshTail } from "./lcd-fresh-tail-bound.js"; +import { computeFreshTailCapChars, boundFreshTailMessages, boundProtectedFreshTail, boundFreshTailTotalToResidual } from "./lcd-fresh-tail-bound.js"; import { factoredMessageTokens } from "./factored-message-tokens.js"; import { LCD_FRESH_TAIL_MAX_TOOL_RESULT_CHARS } from "./constants.js"; @@ -238,3 +238,22 @@ describe("boundProtectedFreshTail — instrumentation + the live OpenAI growth r expect((warn![0] as { errorKind: string }).errorKind).toBe("resource"); }); }); + +describe("boundFreshTailTotalToResidual protected request", () => { + it("keeps the originating user request while dropping completed tool segments", () => { + const request = { role: "user", content: "produce the report" } as AgentMessage; + const tail = [ + request, + { role: "assistant", content: [{ type: "toolCall", id: "old", name: "report", arguments: {} }] }, + { role: "toolResult", toolCallId: "old", content: [{ type: "text", text: "x".repeat(8_000) }] }, + { role: "assistant", content: [{ type: "toolCall", id: "new", name: "report", arguments: {} }] }, + { role: "toolResult", toolCallId: "new", content: [{ type: "text", text: "done" }] }, + ] as unknown as AgentMessage[]; + + const bounded = boundFreshTailTotalToResidual(tail, 100, 0); + + expect(bounded).toContain(request); + expect(JSON.stringify(bounded)).not.toContain('"id":"old"'); + expect(JSON.stringify(bounded)).toContain('"id":"new"'); + }); +}); diff --git a/packages/agent/src/context-engine/lcd-fresh-tail-bound.ts b/packages/agent/src/context-engine/lcd-fresh-tail-bound.ts index 3f4c60a581..5cb5f2bd32 100644 --- a/packages/agent/src/context-engine/lcd-fresh-tail-bound.ts +++ b/packages/agent/src/context-engine/lcd-fresh-tail-bound.ts @@ -216,9 +216,10 @@ function roleOf(m: AgentMessage): string | undefined { * * Operates on the ALREADY per-message-bounded fresh tail (call AFTER * boundFreshTailMessages) so a single oversized message the per-message guard shrank - * to fit is counted at its bounded size, NOT dropped. ALWAYS keeps the LAST step (the current - * live turn ships even when it alone exceeds the residual — the pre-flight then degrades - * it honestly as oversized_input, never silently). A STEP is a leading non-`toolResult` + * to fit is counted at its bounded size, not dropped. It always keeps the protected + * originating-request segment and the newest segment; when those alone exceed the + * residual, the pre-flight reports exhaustion rather than dispatching without the request. + * A STEP is a leading non-`toolResult` * message plus its immediately-following `toolResult`s (mirrors * lcd-budget-eviction.groupIntoSteps / freshTailBoundaryIndex) so a tool_use/tool_result * pair is never split. Pure: reads the input, returns a NEW array (or the same ref when @@ -226,11 +227,13 @@ function roleOf(m: AgentMessage): string | undefined { * * @param freshTail - the per-message-bounded fresh-tail messages ({@link boundFreshTailMessages} output). * @param residualTokens - the room the protected tail may occupy (≈ window − S − floorHeadroom − preamble). - * @returns the newest contiguous fresh-tail steps that fit (≥ the last step). + * @param protectedMessageIndex - message whose segment must remain in the bounded tail. + * @returns the protected segment plus the newest suffix that fits. */ export function boundFreshTailTotalToResidual( freshTail: AgentMessage[], residualTokens: number, + protectedMessageIndex?: number, ): AgentMessage[] { if (freshTail.length === 0) return freshTail; // Step START indices within the fresh tail (a step begins at a non-toolResult msg). @@ -253,6 +256,33 @@ export function boundFreshTailTotalToResidual( } return sum; }; + const protectedStep = protectedMessageIndex === undefined + ? -1 + : stepStarts.findIndex((start, index) => { + const end = stepStarts[index + 1] ?? freshTail.length; + return protectedMessageIndex >= start && protectedMessageIndex < end; + }); + if (protectedStep >= 0 && tokensFrom(0) > residualTokens) { + const protectedStart = stepStarts[protectedStep]!; + const protectedEnd = stepStarts[protectedStep + 1] ?? freshTail.length; + const protectedSlice = freshTail.slice(protectedStart, protectedEnd); + const protectedTokens = protectedSlice.reduce( + (sum, message) => sum + factoredMessageTokens(message), + 0, + ); + if (protectedStep === stepStarts.length - 1) return protectedSlice; + let suffixStep = protectedStep + 1; + while ( + suffixStep < stepStarts.length - 1 + && protectedTokens + tokensFrom(stepStarts[suffixStep]!) > residualTokens + ) { + suffixStep++; + } + return [ + ...protectedSlice, + ...freshTail.slice(stepStarts[suffixStep]!), + ]; + } // Advance past the oldest steps while the remaining tail exceeds the residual; NEVER // drop the last step (stepStarts.at(-1) is the current turn — always kept). let s = 0; @@ -294,6 +324,7 @@ export function boundProtectedFreshTail( logger: ComisLogger; agentId: string | undefined; sessionKey: string | undefined; + protectedMessageIndex?: number; }, ): AgentMessage[] { if (!isFinite(ctx.effectiveWindow) || freshTail.length <= 1) return freshTail; @@ -303,7 +334,11 @@ export function boundProtectedFreshTail( ctx.effectiveWindow - ctx.systemTokens - floorHeadroom - ctx.freshTailPreambleTokens, ); const before = freshTail.reduce((s, m) => s + factoredMessageTokens(m), 0); - const bounded = boundFreshTailTotalToResidual(freshTail, residual); + const bounded = boundFreshTailTotalToResidual( + freshTail, + residual, + ctx.protectedMessageIndex, + ); const after = bounded.reduce((s, m) => s + factoredMessageTokens(m), 0); // INSTRUMENTATION: log the EXACT trim mechanics on EVERY call so a live @@ -358,6 +393,10 @@ export function boundProtectedFreshTail( freshTailFactoredBefore: before, freshTailFactoredAfter: after, // the bounded total — should be ≤ residual when trimmable fitsResidual, + protectedMessageRetained: + ctx.protectedMessageIndex === undefined + ? undefined + : bounded.includes(freshTail[ctx.protectedMessageIndex]!), agentId: ctx.agentId, sessionKey: ctx.sessionKey, }; diff --git a/packages/agent/src/context-engine/lcd-preflight.test.ts b/packages/agent/src/context-engine/lcd-preflight.test.ts index 7a14c9d83a..7cc7f2dde1 100644 --- a/packages/agent/src/context-engine/lcd-preflight.test.ts +++ b/packages/agent/src/context-engine/lcd-preflight.test.ts @@ -212,6 +212,26 @@ describe("capped-window provenance in the exhaustion throw and WARN", () => { expect(p.sessionKey).toBe("t1:u1:c1"); }); + it("emits originating-request retention and trim evidence", () => { + const emit = vi.fn(); + const deps = makeDeps({ + eventBus: { emit } as unknown as ContextEngineDeps["eventBus"], + }); + runPreflightFitCheck(deps, 32_000, [], 0, [], "none", undefined, { + effective: 6, + configured: 8, + originatingRequestRetained: true, + freshTailTrimmedCount: 4, + }); + expect(emit).toHaveBeenCalledWith( + "context:budget_computed", + expect.objectContaining({ + originatingRequestRetained: true, + freshTailTrimmedCount: 4, + }), + ); + }); + it("emits a downshifted-verdict budget event when the thinking governor fires", () => { const emit = vi.fn(); const deps = makeDeps({ diff --git a/packages/agent/src/context-engine/lcd-preflight.ts b/packages/agent/src/context-engine/lcd-preflight.ts index 9dab6b112b..f1d27e7ff2 100644 --- a/packages/agent/src/context-engine/lcd-preflight.ts +++ b/packages/agent/src/context-engine/lcd-preflight.ts @@ -82,7 +82,12 @@ export function runPreflightFitCheck( freshTail: AgentMessage[], reasoningStyle: "none" | "native", capInfo?: ContextWindowCapInfo, - freshTailStepBound?: { effective: number; configured: number }, + freshTailStepBound?: { + effective: number; + configured: number; + originatingRequestRetained?: boolean; + freshTailTrimmedCount?: number; + }, ): number { // Emit effectiveWindow callback so the caller can clamp max_tokens dynamically. deps.onEffectiveWindow?.(effectiveWindow); @@ -155,13 +160,17 @@ export function runPreflightFitCheck( assembledInputTokens: assembled, outputHeadroom: headroom, verdict, - // The STEP bound on the verbatim tail. Threaded so `explain` can say - // "the originating request slid out of the protected tail" instead of a - // clean-looking "fits" (comis-moshe 2026-07-26). + // The verbatim-tail bound and direct coverage evidence used by `explain`. ...(freshTailStepBound !== undefined ? { freshTailSteps: freshTailStepBound.effective, freshTailStepsConfigured: freshTailStepBound.configured, + ...(freshTailStepBound.originatingRequestRetained === undefined + ? {} + : { originatingRequestRetained: freshTailStepBound.originatingRequestRetained }), + ...(freshTailStepBound.freshTailTrimmedCount === undefined + ? {} + : { freshTailTrimmedCount: freshTailStepBound.freshTailTrimmedCount }), } : {}), }); diff --git a/packages/agent/src/executor/executor-response-filter.test.ts b/packages/agent/src/executor/executor-response-filter.test.ts index ba51e6b1f8..c2b96c924b 100644 --- a/packages/agent/src/executor/executor-response-filter.test.ts +++ b/packages/agent/src/executor/executor-response-filter.test.ts @@ -301,7 +301,7 @@ describe("recoverEmptyFinalResponse — tool-call synthesis", () => { stopReason: "toolUse", timestamp: 2, }, - { role: "toolResult", toolCallId: "tc1", content: [{ type: "text", text: "OK" }], timestamp: 3 }, + { role: "toolResult", toolCallId: "tc1", content: [{ type: "text", text: "OK" }], isError: false, timestamp: 3 }, { role: "toolResult", toolCallId: "tc2", content: [{ type: "text", text: "OK" }], timestamp: 4 }, { role: "toolResult", toolCallId: "tc3", content: [{ type: "text", text: "OK" }], timestamp: 5 }, { role: "assistant", content: [], stopReason: "stop", timestamp: 6 }, @@ -331,7 +331,7 @@ describe("recoverEmptyFinalResponse — tool-call synthesis", () => { stopReason: "toolUse", timestamp: 2, }, - { role: "toolResult", toolCallId: "tc1", content: [{ type: "text", text: "OK" }], timestamp: 3 }, + { role: "toolResult", toolCallId: "tc1", content: [{ type: "text", text: "OK" }], isError: false, timestamp: 3 }, { role: "toolResult", toolCallId: "tc2", content: [{ type: "text", text: "OK" }], timestamp: 4 }, { role: "toolResult", toolCallId: "tc3", content: [{ type: "text", text: "OK" }], timestamp: 5 }, { role: "assistant", content: [], stopReason: "stop", timestamp: 6 }, @@ -391,7 +391,7 @@ describe("recoverEmptyFinalResponse — tool-call synthesis", () => { stopReason: "toolUse", timestamp: 2, }, - { role: "toolResult", toolCallId: "tc1", content: [{ type: "text", text: "OK" }], timestamp: 3 }, + { role: "toolResult", toolCallId: "tc1", isError: false, content: [{ type: "text", text: "OK" }], timestamp: 3 }, { role: "toolResult", toolCallId: "tc2", content: [{ type: "text", text: "OK" }], timestamp: 4 }, { role: "toolResult", toolCallId: "tc3", content: [{ type: "text", text: "OK" }], timestamp: 5 }, { role: "assistant", content: [], stopReason: "stop", timestamp: 6 }, @@ -490,7 +490,7 @@ describe("recoverEmptyFinalResponse — tool-call synthesis", () => { stopReason: "toolUse", timestamp: 2, }, - { role: "toolResult", toolCallId: "tc1", content: [{ type: "text", text: "OK" }], timestamp: 3 }, + { role: "toolResult", toolCallId: "tc1", isError: false, content: [{ type: "text", text: "OK" }], timestamp: 3 }, { role: "toolResult", toolCallId: "tc2", content: [{ type: "text", text: "OK" }], timestamp: 4 }, { role: "assistant", content: [], stopReason: "stop", timestamp: 5 }, ], @@ -907,7 +907,13 @@ describe("empty-turn recovery does not narrate an already-delivered reply", () = stopReason: "toolUse", timestamp: 2, }, - { role: "toolResult", toolCallId: "tc1", content: [{ type: "text", text: "OK" }], timestamp: 3 }, + { + role: "toolResult", + toolCallId: "tc1", + isError: false, + content: [{ type: "text", text: "OK" }], + timestamp: 3, + }, { role: "assistant", content: [], stopReason: "stop", timestamp: 4 }, ] as never, logger: mockLogger(), @@ -925,6 +931,24 @@ describe("empty-turn recovery does not narrate an already-delivered reply", () = expect(recover({ action: "reply", channel_type: "telegram", channel_id: "1", text: "answer", message_id: "9" })).toBe(""); }); + it("still recovers when the message delivery tool failed", () => { + const out = recoverEmptyFinalResponse({ + extractedResponse: "", + textEmitted: true, + messages: [ + { role: "user", content: "send it" }, + { + role: "assistant", + content: [{ type: "toolCall", id: "tc-failed", name: "message", arguments: { action: "send", text: "answer" } }], + }, + { role: "toolResult", toolCallId: "tc-failed", isError: true, content: [{ type: "text", text: "delivery failed" }] }, + ], + logger: mockLogger(), + userMessageIndex: 0, + }); + expect(out).toContain("tool-call summary recovered"); + }); + it("STILL synthesizes when the batch delivered NO words (a react is not a reply)", () => { expect(recover({ action: "react", channel_type: "telegram", channel_id: "1", emoji: "\u{1F44D}" })) .toContain("tool-call summary recovered"); diff --git a/packages/agent/src/executor/executor-response-filter.ts b/packages/agent/src/executor/executor-response-filter.ts index be773380b0..02c70eab3e 100644 --- a/packages/agent/src/executor/executor-response-filter.ts +++ b/packages/agent/src/executor/executor-response-filter.ts @@ -341,12 +341,24 @@ function extractVisibleText(content: any[]): string | undefined { */ function deliveredUserFacingText(messages: readonly any[], lowerBound: number): boolean { const DELIVERING_ACTIONS = new Set(["send", "reply"]); + const successfulToolCallIds = new Set(); + for (let i = lowerBound; i < messages.length; i++) { + const msg = messages[i]; // eslint-disable-line security/detect-object-injection + if ( + msg?.role === "toolResult" + && typeof msg.toolCallId === "string" + && msg.isError === false + ) { + successfulToolCallIds.add(msg.toolCallId); + } + } for (let i = lowerBound; i < messages.length; i++) { const msg = messages[i]; // eslint-disable-line security/detect-object-injection if (msg?.role !== "assistant" || !Array.isArray(msg.content)) continue; for (const block of msg.content) { if (block?.type !== "toolCall" && block?.type !== "tool_use") continue; if (block?.name !== "message") continue; + if (typeof block.id !== "string" || !successfulToolCallIds.has(block.id)) continue; const args: Record | undefined = (block?.input && typeof block.input === "object" ? block.input : undefined) ?? (block?.arguments && typeof block.arguments === "object" ? block.arguments : undefined); diff --git a/packages/agent/src/executor/pi-executor/pi-executor.ts b/packages/agent/src/executor/pi-executor/pi-executor.ts index 2f09cd3290..2a2b6a9e60 100644 --- a/packages/agent/src/executor/pi-executor/pi-executor.ts +++ b/packages/agent/src/executor/pi-executor/pi-executor.ts @@ -1596,6 +1596,7 @@ async function runSessionLocked( breaker: toolRetryBreaker, ...(deps.logger === undefined ? {} : { logger: deps.logger }), ...(deps.agentId === undefined ? {} : { agentId: deps.agentId }), + sessionKey: formattedKey, }); } const failedToolRedirects = new Map(); diff --git a/packages/agent/src/safety/background-failure-attribution.test.ts b/packages/agent/src/safety/background-failure-attribution.test.ts index deeaebc8f6..5bd75aaee2 100644 --- a/packages/agent/src/safety/background-failure-attribution.test.ts +++ b/packages/agent/src/safety/background-failure-attribution.test.ts @@ -106,6 +106,22 @@ describe("attributeBackgroundFailuresToOriginatingTool", () => { expect(breaker.recordResult).toHaveBeenCalledTimes(1); }); + it("ignores failures from another session and dispatch redeliveries", () => { + const bus = fakeBus(); + const breaker = { recordResult: vi.fn() }; + attributeBackgroundFailuresToOriginatingTool({ + eventBus: bus as never, + breaker, + agentId: "a", + sessionKey: "session-a", + }); + bus.emit("background_task:failed", { toolName: "t", error: "x", agentId: "a", sessionKey: "session-b" }); + bus.emit("background_task:failed", { toolName: "t", error: "x", agentId: "a", sessionKey: "session-a", dispatchRedelivery: true }); + expect(breaker.recordResult).not.toHaveBeenCalled(); + bus.emit("background_task:failed", { toolName: "t", error: "x", agentId: "a", sessionKey: "session-a" }); + expect(breaker.recordResult).toHaveBeenCalledTimes(1); + }); + it("unsubscribes — the listener must not outlive the execution that owns the breaker", () => { const bus = fakeBus(); const breaker = { recordResult: vi.fn() }; diff --git a/packages/agent/src/safety/background-failure-attribution.ts b/packages/agent/src/safety/background-failure-attribution.ts index b86721efe5..85dce28568 100644 --- a/packages/agent/src/safety/background-failure-attribution.ts +++ b/packages/agent/src/safety/background-failure-attribution.ts @@ -69,6 +69,7 @@ export interface BackgroundFailureAttributionDeps { readonly logger?: ComisLogger; /** Scope to one agent; undefined attributes every agent's failures. */ readonly agentId?: string; + readonly sessionKey?: string; } /** @@ -89,12 +90,14 @@ export function attributeBackgroundFailuresToOriginatingTool( error?: string; agentId?: string; taskId?: string; + sessionKey?: string; + dispatchRedelivery?: boolean; }): void => { + if (p.dispatchRedelivery === true) return; const toolName = p.toolName; if (toolName === undefined || toolName.length === 0) return; - if (deps.agentId !== undefined && p.agentId !== undefined && p.agentId !== deps.agentId) { - return; - } + if (deps.agentId !== undefined && p.agentId !== deps.agentId) return; + if (deps.sessionKey !== undefined && p.sessionKey !== deps.sessionKey) return; // A poller task failing is not the poller's fault either — and it is never // the originating tool, so it would be meaningless to count. if (toolName === BACKGROUND_POLLER_TOOL) return; @@ -105,6 +108,7 @@ export function attributeBackgroundFailuresToOriginatingTool( toolName, taskId: p.taskId, agentId: p.agentId, + sessionKey: p.sessionKey, hint: "counted a background task failure against the tool that launched it — without this the tool reports success on every launch (auto-backgrounding) and its breaker never trips", }, diff --git a/packages/core/src/activity/activity-event.ts b/packages/core/src/activity/activity-event.ts index 2b88c94bd7..674da715bf 100644 --- a/packages/core/src/activity/activity-event.ts +++ b/packages/core/src/activity/activity-event.ts @@ -56,7 +56,7 @@ export const ActivityEventSchema = z.strictObject({ durationMs: z.number().nonnegative().optional(), errorKind: z.enum([ "config", "network", "auth", "validation", "precondition", - "timeout", "resource", "dependency", "internal", "platform", + "timeout", "resource", "dependency", "internal", "platform", "sandbox_unavailable", ]).optional() satisfies z.ZodType, // --- rendering hints (advisory; not authoritative) -------------------- diff --git a/packages/core/src/activity/turn-outcome.ts b/packages/core/src/activity/turn-outcome.ts index ca8b8a1680..7e86756161 100644 --- a/packages/core/src/activity/turn-outcome.ts +++ b/packages/core/src/activity/turn-outcome.ts @@ -82,15 +82,11 @@ export type TurnOutcome = */ reason?: string; /** - * Present when the final answer WAS fully delivered despite the execution - * error — the evidence the coordinator needs to reclassify this turn to - * `success_with_recovered_failures` (answer shown without a failure pill; - * the failure preserved on the outcome and `activity:turn_finalized`). - * Live shape: a backgrounded report timed out (execution lifecycle - * "error") while a later retry delivered the artifact — the user got - * their file with a "❌ dependency" pill above it. NEVER set for a - * resource abort (`reason` present): a stopped run must render as - * stopped even when partial text delivered. + * Present when the final answer was fully delivered despite an execution + * error. The coordinator reclassifies only when the activity timeline also + * proves that each failed tool later completed successfully. Never set for + * a resource abort (`reason` present): a stopped run must render as stopped + * even when partial text delivered. */ delivery?: FinalDeliveryReceipt } | { kind: "silent"; reason: "SILENT" | "HEARTBEAT_OK" | "NO_REPLY" } diff --git a/packages/core/src/api-contracts/incident-report-sections.ts b/packages/core/src/api-contracts/incident-report-sections.ts index 0268962af6..dbeb4ea190 100644 --- a/packages/core/src/api-contracts/incident-report-sections.ts +++ b/packages/core/src/api-contracts/incident-report-sections.ts @@ -53,17 +53,15 @@ export const IncidentContextBudgetSchema = z.object({ verdict: z.enum(["fits", "downshifted", "exhausted"]), /** * Trailing conversation STEPS kept verbatim this turn, EFFECTIVE (post-clamp). - * - * The verbatim tail is bounded by step count, not tokens: a turn with more - * tool round-trips than this slides the user's ORIGINATING request out of - * verbatim context — while `verdict` still reads "fits". Live, that produced - * an agent that apologized for work the user had explicitly requested, with - * 91% of the window unused (comis-moshe 2026-07-26). Optional (additive). */ freshTailSteps: z.number().optional(), /** The configured `contextEngine.freshTailTurns`. A value BELOW this means the * operator's knob was clamped — see {@link freshTailSteps}. */ freshTailStepsConfigured: z.number().optional(), + /** Whether the originating request survived the residual trim. */ + originatingRequestRetained: z.boolean().optional(), + /** Number of fresh-tail messages removed by the residual trim. */ + freshTailTrimmedCount: z.number().int().nonnegative().optional(), }); /** The per-call context budget equation (see {@link IncidentContextBudgetSchema}). */ diff --git a/packages/core/src/event-bus/events-infra.ts b/packages/core/src/event-bus/events-infra.ts index e35a3a7130..6facda8ec7 100644 --- a/packages/core/src/event-bus/events-infra.ts +++ b/packages/core/src/event-bus/events-infra.ts @@ -673,6 +673,7 @@ export interface InfraEvents { taskId: string; toolName: string; error: string; + errorKind: ErrorKind; /** * The TASK's whole lifespan — promote-time to terminal commit. NOT the * duration of the underlying tool call. @@ -705,10 +706,8 @@ export interface InfraEvents { * The ORIGINATING tool call's id, captured at promote time. * * The activity card keys a tool's lifecycle on `tool:`; without - * this field the terminal event could not close the activity it belongs to, - * so a backgrounded tool's card either froze on "running" or (worse) had - * already been closed "completed" at hand-off. Optional: tasks recovered - * from a pre-upgrade on-disk record have none. + * this field the terminal event could not close the activity it belongs to. + * It is omitted when the promotion source has no tool-call correlation. */ toolCallId?: string; /** Formatted session key captured at promote time (activity dispatch requires it). */ diff --git a/packages/core/src/event-bus/events-messaging.ts b/packages/core/src/event-bus/events-messaging.ts index 01a3074b0b..62e5d98525 100644 --- a/packages/core/src/event-bus/events-messaging.ts +++ b/packages/core/src/event-bus/events-messaging.ts @@ -442,24 +442,14 @@ export interface MessagingEvents { outputHeadroom: number; /** Fit-check outcome (closed union — never an open string). */ verdict: "fits" | "downshifted" | "exhausted"; - /** - * The EFFECTIVE number of trailing conversation STEPS kept verbatim this - * turn (post-clamp), and the operator-CONFIGURED value it was clamped from. - * - * The fresh tail is bounded by STEP COUNT, not tokens — so on a long tool - * loop the user's originating request slides out of verbatim context while - * `verdict` still reads "fits" and the window sits nearly empty. That is - * exactly what happened live (comis-moshe 2026-07-26): the agent apologized - * for work the user HAD requested, with 87,740 of 1,000,000 tokens in use - * and `droppedCount:0`. Without these two numbers the report shows a clean - * "fits" and the slide is invisible. - * - * Optional (additive) — emitters that cannot resolve the step bound omit - * both rather than reporting a wrong number. - */ + /** The effective number of trailing conversation steps kept verbatim. */ freshTailSteps?: number; /** The configured `contextEngine.freshTailTurns`. See {@link freshTailSteps}. */ freshTailStepsConfigured?: number; + /** Whether the originating request survived the residual trim. */ + originatingRequestRetained?: boolean; + /** Number of fresh-tail messages removed by the residual trim. */ + freshTailTrimmedCount?: number; }; /** A non-Latin search returned zero hits on a CLEANLY-executed lane. diff --git a/packages/core/src/security/secret-detection.test.ts b/packages/core/src/security/secret-detection.test.ts index 22b44a8c99..519953dc9f 100644 --- a/packages/core/src/security/secret-detection.test.ts +++ b/packages/core/src/security/secret-detection.test.ts @@ -96,26 +96,8 @@ describe("isSecretFieldName — superset", () => { } }); - // --------------------------------------------------------------------------- - // The `.*token` WILDCARD over-match. - // - // `range_token` — an MCP tool argument holding a plain enum value — matched - // `/^.*token$/i`, so LCD/session persistence rewrote it to "[REDACTED]". The - // model then read the placeholder back out of its own replay context and sent - // the literal string "[REDACTED]" to the server, which rejected it - // (`invalid_enum_value received:"[REDACTED]"`). The agent reported that its own - // argument was "exposed to me redacted" and abandoned the cheap query path for a - // far more expensive one that then timed out repeatedly. - // - // The fix is an EXACT-NAME exception set, never a loosening of the pattern — - // every entry below is a non-credential whose name merely ends in "token(s)". - // The negative matrix that follows is the load-bearing half. - // --------------------------------------------------------------------------- - - it("does NOT flag non-credential names the .*token wildcard over-matches", () => { + it("does not flag token-count names that do not end in singular token", () => { for (const name of [ - "range_token", // the live case: an enum selector ("last_week") - "rangeToken", "max_tokens", // provider request knob "maxTokens", "num_tokens", @@ -135,7 +117,20 @@ describe("isSecretFieldName — superset", () => { } }); - it("STILL flags every credential-bearing token name (the exception set must not leak)", () => { + it("keeps the global detector conservative for ambiguous token fields", () => { + expect(isSecretFieldName("range_token")).toBe(true); + expect(isSecretFieldName("rangeToken")).toBe(true); + expect(scanForSecrets({ + plugins: { vendor: { config: { range_token: "shortsecret" } } }, + })).toEqual([ + expect.objectContaining({ + path: "plugins.vendor.config.range_token", + reason: "secret-field", + }), + ]); + }); + + it("flags credential-bearing token names", () => { for (const name of [ "token", "botToken", @@ -193,7 +188,6 @@ describe("isSecretFieldName — superset", () => { "max_tokens_to_sample", "token_count_by_model", "tokenizer_config", - "range_token", "primary_key", // database vocabulary — `key` needs a credential qualifier "foreign_key", "keyboard_layout", diff --git a/packages/core/src/security/secret-detection.ts b/packages/core/src/security/secret-detection.ts index 92bf8a3e73..336a6f11c4 100644 --- a/packages/core/src/security/secret-detection.ts +++ b/packages/core/src/security/secret-detection.ts @@ -230,61 +230,6 @@ const SECRET_FIELD_NAMES: ReadonlySet = new Set([ "api-key", ]); -/** - * Exact (lowercased) field names the `.*token` WILDCARD in - * {@link SECRET_FIELD_PATTERN} over-matches but which are NOT credentials. - * - * ⚠ EXACT names only — never a pattern. A pattern exception here would be a leak - * class (`.*_token` would admit `api_token`, `bot_token`, `refresh_token`). Each - * entry is a name whose semantics are a COUNT, a LIMIT, or a SELECTOR, and every - * one is pinned by the negative matrix in `secret-detection.test.ts` alongside the - * credential names that must keep matching. - * - * Why this exists: an MCP tool argument named `range_token`, holding a plain enum - * value, matched `.*token`, so LCD + session persistence rewrote it to the - * redaction placeholder. The model then read that placeholder back out of its own - * replay context and sent the literal string "[REDACTED]" to the server, which - * rejected it (`invalid_enum_value received:"[REDACTED]"`). Redacting a - * non-secret is not a safe default — it silently corrupts both the tool contract - * and the model's own history. - * - * Adding an entry is a security review: it must be a name that CANNOT carry a - * credential in any integration, not merely one that does not today. - */ -const NON_SECRET_TOKEN_FIELD_NAMES: ReadonlySet = new Set([ - // Range/window SELECTORS (an enum or date-range shorthand, never a credential). - "range_token", - "rangetoken", - // Token COUNTS + LIMITS (provider accounting + request knobs). - "tokens", - "max_tokens", - "maxtokens", - "min_tokens", - "mintokens", - "num_tokens", - "numtokens", - "token_count", - "tokencount", - "token_limit", - "tokenlimit", - "input_tokens", - "inputtokens", - "output_tokens", - "outputtokens", - "total_tokens", - "totaltokens", - "prompt_tokens", - "prompttokens", - "completion_tokens", - "completiontokens", - "cache_read_input_tokens", - "cache_creation_input_tokens", - "cached_tokens", - "cachedtokens", - // The tokenizer itself is a component name, never a credential. - "tokenizer", -]); - // ── Keyword-boundary matcher (closes the end-anchor hole) ── /** @@ -359,19 +304,15 @@ function hasCredentialSegments(name: string): boolean { /** * True if a FIELD NAME implies a secret: * `SECRET_FIELD_PATTERN` ∪ the header set ∪ the keyword-BOUNDARY matcher, - * case-insensitive, MINUS the audited {@link NON_SECRET_TOKEN_FIELD_NAMES} - * over-match exceptions. + * case-insensitive. * - * Order matters: the exception set is consulted FIRST and only ever REMOVES a - * match — an exact header name (`x-auth-token`) is not in the exception set, - * so the header superset is unaffected. The boundary matcher closes the + * The boundary matcher closes the * end-anchor hole (a credential keyword with a SUFFIX after it was invisible * to the `$`-anchored pattern) without the wildcard widening that would * false-redact counting vocabulary. */ export function isSecretFieldName(name: string): boolean { const lower = name.toLowerCase(); - if (NON_SECRET_TOKEN_FIELD_NAMES.has(lower)) return false; return ( SECRET_FIELD_PATTERN.test(name) || SECRET_FIELD_NAMES.has(lower) diff --git a/packages/daemon/src/api/obs-handlers/obs-explain-fresh-tail-verdict.test.ts b/packages/daemon/src/api/obs-handlers/obs-explain-fresh-tail-verdict.test.ts index 17a9fb5e6f..bbbfca20b5 100644 --- a/packages/daemon/src/api/obs-handlers/obs-explain-fresh-tail-verdict.test.ts +++ b/packages/daemon/src/api/obs-handlers/obs-explain-fresh-tail-verdict.test.ts @@ -1,9 +1,8 @@ // SPDX-License-Identifier: Apache-2.0 import { describe, it, expect } from "vitest"; import type { IncidentSignals } from "@comis/core"; -import { freshTailClampedVerdict } from "./obs-explain-fresh-tail-verdict.js"; +import { freshTailOriginLostVerdict } from "./obs-explain-fresh-tail-verdict.js"; -/** The live budget shape from comis-moshe 2026-07-26, seq 293 (the false apology). */ function signals(over: Record = {}): IncidentSignals { return { contextBudget: { @@ -19,46 +18,55 @@ function signals(over: Record = {}): IncidentSignals { verdict: "fits", freshTailSteps: 6, freshTailStepsConfigured: 8, + originatingRequestRetained: false, + freshTailTrimmedCount: 3, ...over, }, } as unknown as IncidentSignals; } -describe("freshTailClampedVerdict", () => { - it("fires when the effective step bound is below the configured freshTailTurns", () => { - const v = freshTailClampedVerdict(signals()); +describe("freshTailOriginLostVerdict", () => { + it("fires when the budget record proves the originating request was lost", () => { + const v = freshTailOriginLostVerdict(signals()); expect(v).not.toBeNull(); - expect(v!.code).toBe("fresh_tail_clamped"); + expect(v!.code).toBe("fresh_tail_origin_lost"); }); it("names BOTH numbers and the knob (so the operator does not have to grep DEBUG)", () => { - const v = freshTailClampedVerdict(signals())!; + const v = freshTailOriginLostVerdict(signals())!; expect(v.detail).toContain("contextEngine.freshTailTurns"); expect(v.detail).toContain("6"); expect(v.detail).toContain("8"); }); - it("states that the token budget was NOT the constraint (the 'fits' trap)", () => { - const v = freshTailClampedVerdict(signals())!; + it("reports actual trim evidence and window use", () => { + const v = freshTailOriginLostVerdict(signals())!; // 87,740 / 1,000,000 = 9% — the window was nearly empty while the request slid out. expect(v.detail).toMatch(/9% of the window/); - expect(v.detail).toMatch(/NOT the constraint/i); + expect(v.detail).toMatch(/3 messages were trimmed/i); }); - it("stays silent when the configured value is honored", () => { - expect(freshTailClampedVerdict(signals({ freshTailSteps: 8 }))).toBeNull(); - expect(freshTailClampedVerdict(signals({ freshTailSteps: 12 }))).toBeNull(); + it("stays silent when a clamp occurred but the request was retained", () => { + expect(freshTailOriginLostVerdict(signals({ originatingRequestRetained: true }))).toBeNull(); }); - it("stays silent on a trajectory that predates the signal (both fields absent)", () => { + it("fires on actual loss even when no clamp occurred", () => { + expect(freshTailOriginLostVerdict(signals({ freshTailSteps: 8 }))).not.toBeNull(); + }); + + it("stays silent when direct origin-retention evidence is absent", () => { expect( - freshTailClampedVerdict( - signals({ freshTailSteps: undefined, freshTailStepsConfigured: undefined }), + freshTailOriginLostVerdict( + signals({ + freshTailSteps: undefined, + freshTailStepsConfigured: undefined, + originatingRequestRetained: undefined, + }), ), ).toBeNull(); }); it("stays silent when there is no budget evidence at all", () => { - expect(freshTailClampedVerdict({} as unknown as IncidentSignals)).toBeNull(); + expect(freshTailOriginLostVerdict({} as unknown as IncidentSignals)).toBeNull(); }); }); diff --git a/packages/daemon/src/api/obs-handlers/obs-explain-fresh-tail-verdict.ts b/packages/daemon/src/api/obs-handlers/obs-explain-fresh-tail-verdict.ts index 5a2ec7851e..d4be8a0892 100644 --- a/packages/daemon/src/api/obs-handlers/obs-explain-fresh-tail-verdict.ts +++ b/packages/daemon/src/api/obs-handlers/obs-explain-fresh-tail-verdict.ts @@ -1,65 +1,33 @@ // SPDX-License-Identifier: Apache-2.0 -/** - * The `fresh_tail_clamped` verdict — the operator's verbatim-tail knob was - * silently overridden. - * - * Sibling of the spend/subagent verdicts (subdir line cap). Pure + deterministic: - * same signals in → same verdict out, no LLM. - * - * WHY this exists (comis-moshe 2026-07-26): the verbatim fresh tail is bounded by - * STEP COUNT, and the clamp reduced the operator's configured value to a smaller - * one on every finite window. On a turn with more tool round-trips than the - * effective bound, the user's ORIGINATING request slid out of verbatim context — - * and the agent then apologized for work the user had explicitly asked for. Every - * numeric lens read healthy at that moment: `verdict:"fits"`, `droppedCount:0`, - * 87,740 of 1,000,000 tokens in use. Nothing in `explain` named the slide, so the - * diagnosis required a DEBUG-only `lcd fresh tail sliced verbatim` log line. - * - * @module - */ +/** Deterministic verdict for a recorded loss of the originating request. */ import type { IncidentSignals, IncidentReport } from "@comis/core"; -/** - * Stamp the verdict when the EFFECTIVE verbatim-tail step bound is below the - * operator-CONFIGURED value. - * - * Ranked LAST in the registry: a real tool/breaker/terminal cause is upstream of - * a context-shaping advisory and must out-rank it. It fires on the otherwise - * clean session — exactly the case that previously produced no verdict at all. - * - * @param s - the folded incident signals. - * @returns the root-cause verdict, or `null` when the knob was honored (or the - * trajectory predates the signal, in which case both fields are absent). - */ -export function freshTailClampedVerdict( +export function freshTailOriginLostVerdict( s: IncidentSignals, ): IncidentReport["likelyRootCause"] | null { const b = s.contextBudget; if (b === undefined) return null; + if (b.originatingRequestRetained !== false) return null; const effective = b.freshTailSteps; const configured = b.freshTailStepsConfigured; - if (effective === undefined || configured === undefined) return null; - if (effective >= configured) return null; const windowUsedPct = b.windowTokens > 0 ? Math.round((b.assembledInputTokens / b.windowTokens) * 100) : 0; return { - code: "fresh_tail_clamped", + code: "fresh_tail_origin_lost", detail: - `the verbatim fresh tail kept only ${String(effective)} trailing steps, but ` + - `contextEngine.freshTailTurns is configured as ${String(configured)} — the operator's value was ` + - `clamped. A turn with more than ${String(effective)} tool round-trips slides the user's own ` + - "request out of verbatim context, so the model can answer as though it was never asked. " + + "the context-budget record proves the user's originating request was absent from the " + + `protected fresh tail after ${String(b.freshTailTrimmedCount ?? 0)} messages were trimmed. ` + + (effective !== undefined && configured !== undefined + ? `The effective step bound was ${String(effective)} and contextEngine.freshTailTurns was ${String(configured)}. ` + : "") + `This turn used ${String(b.assembledInputTokens)} of ${String(b.windowTokens)} tokens ` + - `(${String(windowUsedPct)}% of the window), so the eviction budget was NOT the constraint — ` + - "the step bound was.", + `(${String(windowUsedPct)}% of the window).`, suggestedNextSteps: [ - `compare the effective bound (${String(effective)}) against contextEngine.freshTailTurns ` + - `(${String(configured)}) — a persistent gap means the clamp, not the config, is deciding`, - "check whether the turn's tool round-trips exceeded the effective bound (the trajectory's " + - "tool.call records between the user's prompt.submitted and the reply)", + "inspect the context.budget record's originatingRequestRetained and freshTailTrimmedCount fields", + "reduce fixed prompt overhead or the number of retained completed tool segments so the originating request remains protected", ], }; } diff --git a/packages/daemon/src/api/obs-handlers/obs-explain-heuristics.ts b/packages/daemon/src/api/obs-handlers/obs-explain-heuristics.ts index 5207f38eb7..9bd5998e29 100644 --- a/packages/daemon/src/api/obs-handlers/obs-explain-heuristics.ts +++ b/packages/daemon/src/api/obs-handlers/obs-explain-heuristics.ts @@ -75,7 +75,7 @@ import { import { learnedSkillFailingVerdict, synthesisAbstainedVerdict } from "./obs-explain-learning-verdicts.js"; import { spendExceededVerdict } from "./obs-explain-spend-verdict.js"; // NAMED spend verdict (sibling — subdir cap) import { subagentStuckKilledVerdict } from "./obs-explain-subagent-killed-verdict.js"; // health-monitor-killed sub-agent (sibling — subdir cap) -import { freshTailClampedVerdict } from "./obs-explain-fresh-tail-verdict.js"; // verbatim-tail clamp advisory (sibling — subdir cap) +import { freshTailOriginLostVerdict } from "./obs-explain-fresh-tail-verdict.js"; import { backgroundPendingVerdict, backgroundRecoveryVerdict, @@ -582,12 +582,9 @@ export const HEURISTICS: ReadonlyArray<(s: IncidentSignals) => RootCause | null> }; }, - // N) fresh_tail_clamped — DEAD LAST. A context-SHAPING advisory, not a - // terminal cause: every acute verdict above out-ranks it. It fires on the - // session that previously produced NO verdict at all — a "clean" turn whose - // originating request quietly slid out of the verbatim tail because the - // step bound (not the token budget) evicted it. Sibling module (subdir cap). - freshTailClampedVerdict, + // N) fresh_tail_origin_lost — DEAD LAST. A context-shaping advisory, not a + // terminal cause: every acute verdict above out-ranks it. + freshTailOriginLostVerdict, ]; /** Run the ordered registry; first non-null `RootCause` wins, else `null` (clean session). */ diff --git a/packages/daemon/src/daemon.ts b/packages/daemon/src/daemon.ts index a77413b263..19558cd833 100644 --- a/packages/daemon/src/daemon.ts +++ b/packages/daemon/src/daemon.ts @@ -8,7 +8,7 @@ * reusable wiring lives under `./wiring/`. * * @module - */import { assertProactiveFailureIsSupported, proactiveNotArmedLogFields, PROACTIVE_NOT_ARMED_MSG, EMPTY_PROACTIVE_HANDLES } from "./wiring/proactive-degrade.js"; + */import { assertProactiveFailureIsSupported, proactiveNotArmedLogFields, proactiveNotArmedMessage, EMPTY_PROACTIVE_HANDLES } from "./wiring/proactive-degrade.js"; import { bootstrap, @@ -2097,7 +2097,7 @@ async function bootChannels(boot: BootContext): Promise { const msTeamsConversationStore = createSqliteMsTeamsConversationStore(db); // shared memory.db → createMsTeamsPlugin (capture + proactive recovery) // 7.9. Capability-lease layer + ACTIVATION — constructed BEFORE setupTools so the KEPT handle threads capMint + the orchestrate capSocketPath into tool assembly; on `boot` for bootShutdown. cronJobCount binds the bounded-autonomy rate count to the per-agent CronScheduler. durableRuns threads into the jail-leg chokepoint for the _outwardStepIndex allocation. - const { capEndpointHandle, namespacePreflightOk } = await constructCapabilityLayer({ agents, rpcCall, clock: boot.clock, timers: handle.timers, cronJobCount: (agentId) => { try { const jobs = handle.getAgentCronScheduler(agentId).getJobs(); return jobs.ok ? jobs.value.length : 0; } catch { return 0; } }, dataDir: container.config.dataDir || ".", daemonLogger, skillsLogger, workspaceDirs, defaultWorkspaceDir, webSearchKeys: container.secretManager, boundedAutonomyHolder: handle.boundedAutonomyBudgetHolder, leaseManager: handle.sharedLeaseManager, container, mcpClientManager, ...(durableRunStoreEarly ? { durableRuns: durableRunStoreEarly } : {}), ...(outwardLedgerEarly ? { outwardLedger: outwardLedgerEarly } : {}) }); // POPULATES the late-bound budget holder (read by the bridge at turn time) + shares the SAME LeaseManager as the cron-fire mint; the container is passed so the SOCKET chokepoint emits the per-cap audit (audit:event + capability:audited) for jailed tool.invoke calls + const { capEndpointHandle, capEndpointUnavailableReason, namespacePreflightOk } = await constructCapabilityLayer({ agents, rpcCall, clock: boot.clock, timers: handle.timers, cronJobCount: (agentId) => { try { const jobs = handle.getAgentCronScheduler(agentId).getJobs(); return jobs.ok ? jobs.value.length : 0; } catch { return 0; } }, dataDir: container.config.dataDir || ".", daemonLogger, skillsLogger, workspaceDirs, defaultWorkspaceDir, webSearchKeys: container.secretManager, boundedAutonomyHolder: handle.boundedAutonomyBudgetHolder, leaseManager: handle.sharedLeaseManager, container, mcpClientManager, ...(durableRunStoreEarly ? { durableRuns: durableRunStoreEarly } : {}), ...(outwardLedgerEarly ? { outwardLedger: outwardLedgerEarly } : {}) }); // POPULATES the late-bound budget holder (read by the bridge at turn time) + shares the SAME LeaseManager as the cron-fire mint; the container is passed so the SOCKET chokepoint emits the per-cap audit (audit:event + capability:audited) for jailed tool.invoke calls Object.assign(boot, { capEndpointHandle, namespacePreflightOk }); if (capEndpointHandle && sandboxProvider) { // pre-payload wake-gate runner: built after the cap layer (deps from capEndpointHandle), read at fire time @@ -2452,9 +2452,12 @@ async function bootChannels(boot: BootContext): Promise { runtime: handle, adaptersByType, deliveryService, schedulerCorePortBindings: handle.schedulerCorePortBindings, }); - assertProactiveFailureIsSupported(proactive); + assertProactiveFailureIsSupported(proactive, capEndpointUnavailableReason); if (proactive.ok) handle.bindTaskMaintenanceRuntime(proactive.value); - else daemonLogger.error(proactiveNotArmedLogFields(), PROACTIVE_NOT_ARMED_MSG); + else daemonLogger.error( + proactiveNotArmedLogFields(capEndpointUnavailableReason), + proactiveNotArmedMessage(capEndpointUnavailableReason), + ); const { heartbeatRunner, duplicateDetector, coordinator: heartbeatCoordinator } = proactive.ok ? proactive.value : EMPTY_PROACTIVE_HANDLES; // 6.7.0.2. Agent management runtime state const suspendedAgents = new Set(); diff --git a/packages/daemon/src/wiring/proactive-degrade.test.ts b/packages/daemon/src/wiring/proactive-degrade.test.ts new file mode 100644 index 0000000000..49f4b630f8 --- /dev/null +++ b/packages/daemon/src/wiring/proactive-degrade.test.ts @@ -0,0 +1,50 @@ +// SPDX-License-Identifier: Apache-2.0 +import { describe, expect, it } from "vitest"; +import { + assertProactiveFailureIsSupported, + isAutonomyDisabledProactiveMiss, + proactiveNotArmedLogFields, + proactiveNotArmedMessage, +} from "./proactive-degrade.js"; + +const missingCapabilityEndpoint = { + code: "dependency_unavailable", + message: "Missing proactive dependencies: capEndpointHandle (1 of 11 unavailable)", +}; + +describe("proactive scheduler degradation reason", () => { + it("accepts a missing capability endpoint only when autonomy is disabled", () => { + expect(isAutonomyDisabledProactiveMiss( + missingCapabilityEndpoint, + "autonomy_disabled", + )).toBe(true); + expect(isAutonomyDisabledProactiveMiss( + missingCapabilityEndpoint, + "activation_failed", + )).toBe(false); + }); + + it("degrades with activation-specific guidance when endpoint activation fails", () => { + expect(() => assertProactiveFailureIsSupported( + { ok: false, error: missingCapabilityEndpoint }, + "activation_failed", + )).not.toThrow(); + const fields = proactiveNotArmedLogFields("activation_failed"); + expect(fields.hint).toContain("config.dataDir"); + expect(fields.hint).not.toContain("autonomy.enabled"); + expect(proactiveNotArmedMessage("activation_failed")).toContain("activation failed"); + }); + + it("rejects unrelated proactive dependency failures", () => { + expect(() => assertProactiveFailureIsSupported( + { + ok: false, + error: { + code: "dependency_unavailable", + message: "Missing proactive dependencies: deliveryService (1 of 11 unavailable)", + }, + }, + "activation_failed", + )).toThrow("Proactive scheduler activation failed"); + }); +}); diff --git a/packages/daemon/src/wiring/proactive-degrade.ts b/packages/daemon/src/wiring/proactive-degrade.ts index 632fd10333..dad9c40aa4 100644 --- a/packages/daemon/src/wiring/proactive-degrade.ts +++ b/packages/daemon/src/wiring/proactive-degrade.ts @@ -3,12 +3,11 @@ /** * Honest degradation when proactive schedulers cannot be armed. * - * `constructCapabilityLayer` returns `capEndpointHandle: undefined` BY DESIGN - * when no agent has autonomy enabled, and `setupProactiveSchedulers` treats that - * handle as mandatory. The daemon used to throw on the failed Result — so a fully - * supported config completed its entire boot (channels registered, adapter - * polling) and then exited 1, forever. `systemctl is-active` read `active` - * throughout while the box served nothing. + * `constructCapabilityLayer` returns `capEndpointHandle: undefined` when no + * agent has autonomy enabled or when endpoint activation fails, while + * `setupProactiveSchedulers` treats that handle as mandatory. Both supported + * degradation paths keep channels available and report why proactive work is + * unavailable. * * Reachability is the severity: an omitted `autonomy` block defaults to ENABLED, * but writing any sub-key without `enabled: true` — e.g. the documented @@ -17,6 +16,8 @@ * @module */ +import type { CapabilityEndpointUnavailableReason } from "./setup-capability-endpoint-boot.js"; + /** The failure shape a failed `setupProactiveSchedulers` returns. */ interface ProactiveSetupError { readonly code: string; @@ -24,16 +25,30 @@ interface ProactiveSetupError { } /** - * True when the ONLY unmet proactive-scheduler dependency is the capability - * endpoint — i.e. the supported autonomy-disabled configuration, not a - * composition-root regression. + * True when the only unmet proactive-scheduler dependency is a capability + * endpoint omitted because autonomy is disabled. * * @param error - the failed setup Result's error. * @returns whether the daemon may boot without the proactive surface. */ -export function isAutonomyDisabledProactiveMiss(error: ProactiveSetupError): boolean { +export function isAutonomyDisabledProactiveMiss( + error: ProactiveSetupError, + capEndpointUnavailableReason?: CapabilityEndpointUnavailableReason, +): boolean { + return ( + capEndpointUnavailableReason === "autonomy_disabled" + && error.code === "dependency_unavailable" + && /capEndpointHandle \(1 of 11/.test(error.message) + ); +} + +function isCapabilityActivationProactiveMiss( + error: ProactiveSetupError, + capEndpointUnavailableReason?: CapabilityEndpointUnavailableReason, +): boolean { return ( - error.code === "dependency_unavailable" + capEndpointUnavailableReason === "activation_failed" + && error.code === "dependency_unavailable" && /capEndpointHandle \(1 of 11/.test(error.message) ); } @@ -44,9 +59,20 @@ export function isAutonomyDisabledProactiveMiss(error: ProactiveSetupError): boo * * @returns content-free log fields. */ -export function proactiveNotArmedLogFields(): Record { +export function proactiveNotArmedLogFields( + reason?: CapabilityEndpointUnavailableReason, +): Record { + if (reason === "activation_failed") { + return { + submodule: "setup-proactive-schedulers", + errorKind: "config" as const, + hint: + "Cron jobs and the heartbeat are NOT armed for this boot because the capability " + + "endpoint could not activate. Check that config.dataDir is absolute and writable " + + "and that its cap.sock path can be created, then restart the daemon.", + }; + } return { - module: "daemon", submodule: "setup-proactive-schedulers", errorKind: "config" as const, hint: @@ -63,18 +89,30 @@ export function proactiveNotArmedLogFields(): Record { export const PROACTIVE_NOT_ARMED_MSG = "Proactive schedulers not armed: autonomy is disabled for every agent"; +export function proactiveNotArmedMessage( + reason?: CapabilityEndpointUnavailableReason, +): string { + return reason === "activation_failed" + ? "Proactive schedulers not armed: capability endpoint activation failed" + : PROACTIVE_NOT_ARMED_MSG; +} + /** - * Abort boot unless a failed proactive-scheduler setup is the SUPPORTED - * autonomy-disabled case. + * Abort boot unless a failed proactive-scheduler setup has a known capability + * endpoint degradation reason. * * @param proactive - the `setupProactiveSchedulers` Result. * @throws when the failure is a genuine composition-root regression. */ export function assertProactiveFailureIsSupported( proactive: { ok: true } | { ok: false; error: ProactiveSetupError }, + capEndpointUnavailableReason?: CapabilityEndpointUnavailableReason, ): void { if (proactive.ok) return; - if (isAutonomyDisabledProactiveMiss(proactive.error)) return; + if ( + isAutonomyDisabledProactiveMiss(proactive.error, capEndpointUnavailableReason) + || isCapabilityActivationProactiveMiss(proactive.error, capEndpointUnavailableReason) + ) return; throw new Error(`Proactive scheduler activation failed: ${proactive.error.message}`); } diff --git a/packages/daemon/src/wiring/setup-capability-endpoint-boot.test.ts b/packages/daemon/src/wiring/setup-capability-endpoint-boot.test.ts index 7b6bd7a1c2..750c6a7b1b 100644 --- a/packages/daemon/src/wiring/setup-capability-endpoint-boot.test.ts +++ b/packages/daemon/src/wiring/setup-capability-endpoint-boot.test.ts @@ -88,6 +88,7 @@ describe("constructCapabilityLayer autonomy gate + boot preflight", () => { const result = await constructCapabilityLayer(createDeps({})); expect(result.capEndpointHandle).toBeUndefined(); expect(result.capEndpointStop).toBeUndefined(); + expect(result.capEndpointUnavailableReason).toBe("autonomy_disabled"); expect(typeof result.namespacePreflightOk).toBe("boolean"); }); @@ -100,6 +101,7 @@ describe("constructCapabilityLayer autonomy gate + boot preflight", () => { const deps = createDeps(agents); const result = await constructCapabilityLayer(deps); expect(result.capEndpointHandle).toBeUndefined(); + expect(result.capEndpointUnavailableReason).toBe("autonomy_disabled"); // Activation did NOT happen → the active:true INFO is never logged. expect(deps.daemonLogger.info).not.toHaveBeenCalled(); }); @@ -339,6 +341,7 @@ describe("constructCapabilityLayer autonomy gate + boot preflight", () => { const result = await constructCapabilityLayer(deps); expect(result.capEndpointHandle).toBeUndefined(); expect(result.capEndpointStop).toBeUndefined(); + expect(result.capEndpointUnavailableReason).toBe("activation_failed"); // The host preflight still ran (it is a host check, independent of the socket). expect(typeof result.namespacePreflightOk).toBe("boolean"); const warnMock = deps.daemonLogger.warn as unknown as ReturnType; diff --git a/packages/daemon/src/wiring/setup-capability-endpoint-boot.ts b/packages/daemon/src/wiring/setup-capability-endpoint-boot.ts index 308b56a3f6..d6a3b133bd 100644 --- a/packages/daemon/src/wiring/setup-capability-endpoint-boot.ts +++ b/packages/daemon/src/wiring/setup-capability-endpoint-boot.ts @@ -269,16 +269,10 @@ export interface CapabilityLayerHandle { escalate: NotifyFn; } -/** Result of {@link constructCapabilityLayer}: the cap handle + the boot preflight boolean. */ -export interface CapabilityLayerResult { - /** Undefined when NO agent resolves to an autonomy-bearing profile. */ - capEndpointHandle: CapabilityLayerHandle | undefined; - /** - * Shutdown teardown thunk for the cap endpoint socket (stops + unlinks - * cap.sock), or `undefined` when no endpoint was constructed. Threaded into - * `setupShutdown` (mirrors the broker teardown). - */ - capEndpointStop: (() => Promise) | undefined; +/** Closed reason for constructing no capability endpoint. */ +export type CapabilityEndpointUnavailableReason = "autonomy_disabled" | "activation_failed"; + +interface CapabilityLayerResultBase { /** * Host namespace preflight result (the unprivileged-userns + * `--unshare-net` probe, run once at boot). Fed to the SHIPPED degradeAutonomy @@ -299,6 +293,21 @@ export interface CapabilityLayerResult { resolveRootRunId?: RootRunIdResolver; } +/** Result of capability-layer construction and its host preflight. */ +export type CapabilityLayerResult = CapabilityLayerResultBase & ( + | { + capEndpointHandle: CapabilityLayerHandle; + /** Stops the capability endpoint and unlinks its socket. */ + capEndpointStop: () => Promise; + capEndpointUnavailableReason?: never; + } + | { + capEndpointHandle: undefined; + capEndpointStop: undefined; + capEndpointUnavailableReason: CapabilityEndpointUnavailableReason; + } +); + /** The shipped web-search provider keys, read from the secret store (or none). */ function buildWebSearchConfig( keys: CapabilityWebSearchKeys | undefined, @@ -587,7 +596,12 @@ export async function constructCapabilityLayer( .map((a) => resolveAutonomy(a.autonomy)) .find((r) => r.enabled); if (!autonomyBearingConfig) { - return { capEndpointHandle: undefined, capEndpointStop: undefined, namespacePreflightOk }; + return { + capEndpointHandle: undefined, + capEndpointStop: undefined, + capEndpointUnavailableReason: "autonomy_disabled", + namespacePreflightOk, + }; } // Use the daemon-supplied LeaseManager when provided @@ -762,6 +776,11 @@ export async function constructCapabilityLayer( }, "Capability lease layer DEGRADED — cap-socket activation failed; autonomy surface unavailable (daemon continues serving channels)", ); - return { capEndpointHandle: undefined, capEndpointStop: undefined, namespacePreflightOk }; + return { + capEndpointHandle: undefined, + capEndpointStop: undefined, + capEndpointUnavailableReason: "activation_failed", + namespacePreflightOk, + }; } } diff --git a/packages/observability/src/activity/activity-stream.test.ts b/packages/observability/src/activity/activity-stream.test.ts index d951d880d4..0f1f5f347c 100644 --- a/packages/observability/src/activity/activity-stream.test.ts +++ b/packages/observability/src/activity/activity-stream.test.ts @@ -899,6 +899,7 @@ describe("backgrounded tools close on their REAL terminal, not the hand-off (F-A taskId: "task-1", toolName: "mcp__vendor-mcp--vendor_activity_report", error: "MCP error -32001: Request timed out", + errorKind: "timeout", durationMs: 120_000, origin: {} as never, timestamp: 2, @@ -910,7 +911,7 @@ describe("backgrounded tools close on their REAL terminal, not the hand-off (F-A expect(ends).toHaveLength(1); expect(ends[0]!.status).toBe("failed"); expect(ends[0]!.toolCallId).toBe("call-1"); - expect(ends[0]!.errorKind).toBe("dependency"); + expect(ends[0]!.errorKind).toBe("timeout"); }); it("the background_task:completed terminal closes it COMPLETED", () => { @@ -939,6 +940,7 @@ describe("backgrounded tools close on their REAL terminal, not the hand-off (F-A taskId: "task-1", toolName: "report", error: "boom", + errorKind: "dependency", durationMs: 5, origin: {} as never, timestamp: 2, @@ -951,13 +953,14 @@ describe("backgrounded tools close on their REAL terminal, not the hand-off (F-A expect(events.filter((e) => e.phase === "end")).toHaveLength(1); }); - it("a pre-upgrade terminal with NO toolCallId is a no-op (the old close already happened)", () => { + it("a terminal without toolCallId does not close unrelated activity", () => { const { bus, events } = harness(); bus.emit("background_task:failed", { agentId: CTX.agentId, taskId: "task-1", toolName: "report", error: "boom", + errorKind: "dependency", durationMs: 5, origin: {} as never, timestamp: 2, diff --git a/packages/observability/src/activity/activity-stream.ts b/packages/observability/src/activity/activity-stream.ts index 249535bc8c..d57c9c8781 100644 --- a/packages/observability/src/activity/activity-stream.ts +++ b/packages/observability/src/activity/activity-stream.ts @@ -496,9 +496,8 @@ export function createActivityStream(deps: CreateActivityStreamDeps): ActivitySt * Close a BACKGROUNDED tool's activity on its real terminal event. * * The hand-off left the card `running` (see onToolExecuted); this is the - * matching close. Correlation is the promote-time capture on the event — - * a pre-upgrade task record has none, in which case there is no card to - * close (the old hand-off path already closed it) and we do nothing. + * matching close. Correlation is the promote-time capture on the event; + * without it there is no activity card that can be closed safely. * A `dispatchRedelivery` re-emit is a dispatch mechanism, not a second * outcome — never re-close on it. */ @@ -524,7 +523,7 @@ export function createActivityStream(deps: CreateActivityStreamDeps): ActivitySt semanticPhase: failed ? "error" : semanticPhase, toolName: p.toolName, durationMs: p.durationMs, - ...(failed ? { errorKind: "dependency" as const } : {}), + ...(failed ? { errorKind: (p as EventMap["background_task:failed"]).errorKind } : {}), defaultLabel, }); } diff --git a/packages/observability/src/trajectory/event-bus-bridge.test.ts b/packages/observability/src/trajectory/event-bus-bridge.test.ts index 3d2ca6e575..8c41521def 100644 --- a/packages/observability/src/trajectory/event-bus-bridge.test.ts +++ b/packages/observability/src/trajectory/event-bus-bridge.test.ts @@ -1215,6 +1215,7 @@ describe("attachTrajectoryToEventBus -- envelope-only correlation invariant", () taskId: "t-1", toolName: "exec", error: "boom", + errorKind: "dependency", durationMs: 9, origin: { agentId: "default", sessionKey: "k" }, timestamp: 1000, @@ -1602,6 +1603,8 @@ describe("attachTrajectoryToEventBus -- envelope-only correlation invariant", () assembledInputTokens: 31_572, outputHeadroom: 768, verdict: "exhausted", + originatingRequestRetained: false, + freshTailTrimmedCount: 3, }, "context:evicted": { agentId: "agent-1", @@ -3568,6 +3571,8 @@ describe("security + compaction + context + approval bridge", () => { assembledInputTokens: 31_572, outputHeadroom: 768, verdict: "exhausted", + originatingRequestRetained: false, + freshTailTrimmedCount: 3, }); expect(recorder.calls).toHaveLength(1); @@ -3584,6 +3589,8 @@ describe("security + compaction + context + approval bridge", () => { expect(data.assembledInputTokens).toBe(31_572); expect(data.outputHeadroom).toBe(768); expect(data.verdict).toBe("exhausted"); + expect(data.originatingRequestRetained).toBe(false); + expect(data.freshTailTrimmedCount).toBe(3); expect(data.agentId).toBeUndefined(); expect(data.sessionKey).toBeUndefined(); }); @@ -4579,6 +4586,7 @@ describe("background-task dispatch redelivery is not a second failure", () => { taskId: "task-a", toolName: "mcp__vendor-mcp--vendor_activity_report", error: "MCP error -32001: Request timed out", + errorKind: "timeout", durationMs: 120_000, origin, timestamp: 10, diff --git a/packages/observability/src/trajectory/translate-payload.test.ts b/packages/observability/src/trajectory/translate-payload.test.ts index 79e6270bad..055ba65c6b 100644 --- a/packages/observability/src/trajectory/translate-payload.test.ts +++ b/packages/observability/src/trajectory/translate-payload.test.ts @@ -170,11 +170,12 @@ describe("translatePayload — T2.2 background_task lifecycle (F9: now visible o taskId: "t-1", toolName: "exec", error: "secret-looking stack trace", + errorKind: "timeout", durationMs: 9, origin: { agentId: "a1", sessionKey: "k" }, timestamp: 300, }); - expect(data).toEqual({ taskId: "t-1", toolName: "exec", durationMs: 9 }); + expect(data).toEqual({ taskId: "t-1", toolName: "exec", durationMs: 9, errorKind: "timeout" }); expect(JSON.stringify(data)).not.toMatch(/secret-looking stack trace/); }); }); diff --git a/packages/observability/src/trajectory/translate-payload.ts b/packages/observability/src/trajectory/translate-payload.ts index 300e99b0a0..0484c042dc 100644 --- a/packages/observability/src/trajectory/translate-payload.ts +++ b/packages/observability/src/trajectory/translate-payload.ts @@ -357,7 +357,12 @@ export function translatePayload( case "background_task:completed": return { taskId: payload.taskId, toolName: payload.toolName, durationMs: payload.durationMs }; case "background_task:failed": - return { taskId: payload.taskId, toolName: payload.toolName, durationMs: payload.durationMs }; + return { + taskId: payload.taskId, + toolName: payload.toolName, + durationMs: payload.durationMs, + errorKind: payload.errorKind, + }; case "background_task:notified": // The fallback-notice decision — taskId + tool NAME + the notified bool + // closed-union reason ONLY. agentId/sessionKey/traceId/timestamp are @@ -776,6 +781,12 @@ export function translatePayload( ...(payload.freshTailStepsConfigured !== undefined ? { freshTailStepsConfigured: payload.freshTailStepsConfigured } : {}), + ...(payload.originatingRequestRetained !== undefined + ? { originatingRequestRetained: payload.originatingRequestRetained } + : {}), + ...(payload.freshTailTrimmedCount !== undefined + ? { freshTailTrimmedCount: payload.freshTailTrimmedCount } + : {}), budgetedHistoryTokens: payload.budgetedHistoryTokens, keptCount: payload.keptCount, assembledInputTokens: payload.assembledInputTokens, diff --git a/packages/orchestrator/src/execution/activity-turn-coordinator.test.ts b/packages/orchestrator/src/execution/activity-turn-coordinator.test.ts index 390218dbca..56079296c9 100644 --- a/packages/orchestrator/src/execution/activity-turn-coordinator.test.ts +++ b/packages/orchestrator/src/execution/activity-turn-coordinator.test.ts @@ -322,7 +322,8 @@ describe("createActivityTurnCoordinator — delete gate", () => { coord.start(makeCtx()); // Observe a failed event during the turn (a recovered tool retry). - stream.emit(makeEvent({ status: "failed", errorKind: "dependency", phase: "end" })); + stream.emit(makeEvent({ status: "failed", errorKind: "dependency", phase: "end", toolName: "report" })); + stream.emit(makeEvent({ status: "completed", phase: "end", toolName: "report" })); timer.advance(800); // Delivery SUCCEEDED — the turn recovered. The renderer must get the @@ -1278,7 +1279,8 @@ describe("a delivered answer never renders a failure pill (F-ACT-1 layer 4)", () coord.start(makeCtx()); // The backgrounded tool's REAL terminal, observed during the turn. - stream.emit(makeEvent({ status: "failed", errorKind: "dependency", phase: "end" })); + stream.emit(makeEvent({ status: "failed", errorKind: "dependency", phase: "end", toolName: "report" })); + stream.emit(makeEvent({ status: "completed", phase: "end", toolName: "report" })); timer.advance(800); await coord.finalize({ @@ -1296,6 +1298,25 @@ describe("a delivered answer never renders a failure pill (F-ACT-1 layer 4)", () coord.dispose(); }); + it("delivery alone does not recover a terminal execution failure", async () => { + const clock = createFakeClock(5_000); + const { deps, timer, stream, renderer } = makeCoordinatorDeps({ clock }); + const coord = createActivityTurnCoordinator(deps); + coord.start(makeCtx()); + stream.emit(makeEvent({ status: "failed", errorKind: "dependency", phase: "end", toolName: "report" })); + timer.advance(800); + + await coord.finalize({ + kind: "failure", + errorKind: "dependency", + failedEvents: [], + delivery: { deliveredAtMs: clock.now() } as never, + }); + + expect(renderer.finalizeCalls[0]!.outcome.kind).toBe("failure"); + coord.dispose(); + }); + it("failure + delivered evidence but NO observed failed events keeps the truthful failure", async () => { const clock = createFakeClock(5_000); const { deps, renderer } = makeCoordinatorDeps({ clock }); diff --git a/packages/orchestrator/src/execution/activity-turn-coordinator.ts b/packages/orchestrator/src/execution/activity-turn-coordinator.ts index 4c1af51bed..3d2fe825e1 100644 --- a/packages/orchestrator/src/execution/activity-turn-coordinator.ts +++ b/packages/orchestrator/src/execution/activity-turn-coordinator.ts @@ -12,8 +12,8 @@ * injected `TimerPort` (`handle.cancel()` for cancellation — never a raw * timer global), * 3. on `finalize(outcome)` enforces the delete gate: - * • any observed `ActivityEvent{status:"failed"}` on a DELIVERED success - * reclassifies the outcome to `success_with_recovered_failures` — the + * • a delivered success with observed failures reclassifies the outcome + * to `success_with_recovered_failures` — the * renderer's success-shaped cleanup runs (a recovered turn never keeps * a "❌ {errorKind}" pill above its delivered answer) and the failed * events ride the outcome + the `activity:turn_finalized` event as @@ -575,16 +575,24 @@ export function createActivityTurnCoordinator(deps: ActivityTurnCoordinatorDeps) } } - // (1a2) A failure whose answer WAS fully delivered, with the failure - // attributable to observed events, is a SUCCESS WITH RECOVERED FAILURES: + // (1a2) A failure whose answer was fully delivered, with every observed + // failure followed by a successful completion of the same tool, is a + // SUCCESS WITH RECOVERED FAILURES: // the user's chat shows the answer (no failure pill), while the failed // events ride the outcome + `activity:turn_finalized` as evidence. This is - // NOT "delivery succeeded ⇒ success" — with no observed failed events the - // failure evidence would be erased, so that case keeps the truthful - // failure (and gets the named reason below). + // Delivery without matching recovery evidence keeps the failure. if (effective.kind === "failure" && effective.delivery !== undefined) { const failedEvents = events.filter((e) => e.status === "failed"); - if (isNonEmptyEvents(failedEvents)) { + const everyFailureRecovered = failedEvents.length > 0 && failedEvents.every((failed) => { + const failureIndex = events.indexOf(failed); + return events.slice(failureIndex + 1).some((later) => + later.status === "completed" + && later.kind === failed.kind + && later.toolName !== undefined + && later.toolName === failed.toolName, + ); + }); + if (everyFailureRecovered && isNonEmptyEvents(failedEvents)) { effective = { kind: "success_with_recovered_failures", trivial: false, diff --git a/packages/orchestrator/src/execution/execution-pipeline.ts b/packages/orchestrator/src/execution/execution-pipeline.ts index 9052b11642..5bad41e751 100644 --- a/packages/orchestrator/src/execution/execution-pipeline.ts +++ b/packages/orchestrator/src/execution/execution-pipeline.ts @@ -712,10 +712,9 @@ export async function executeAndDeliver( // cannot reclassify an already-delivered turn or trigger inbound fallback. if (coordinatorExecutionOutcome) { // A resource abort (reason set) renders as the truthful stop even when - // partial text delivered. An UNATTRIBUTED lifecycle failure whose final - // answer WAS fully delivered gets the receipt attached, so the - // coordinator can reclassify to success_with_recovered_failures instead - // of pinning a failure marker above a delivered answer (proven live). + // partial text delivered. An unattributed lifecycle failure whose final + // answer was fully delivered gets the receipt attached; the coordinator + // reclassifies only when the activity timeline also proves recovery. await finalizeCoordinator( withDeliveredEvidence( coordinatorExecutionOutcome, diff --git a/packages/skills/src/platform-tools/tools/gateway-tool.test.ts b/packages/skills/src/platform-tools/tools/gateway-tool.test.ts index c0d076bafe..420ef9b38c 100644 --- a/packages/skills/src/platform-tools/tools/gateway-tool.test.ts +++ b/packages/skills/src/platform-tools/tools/gateway-tool.test.ts @@ -1411,5 +1411,35 @@ describe("gated env_set survives its own confirmation (the approval/redaction de action: "env_set" as "read", env_key: "WRONG_KEY", _confirmed: true, pending_action_id: pendingId, } as any); expect((wrong.details as Record).error).toBe("pending_action_key_mismatch"); + + const correct = await tool.execute("call-p10", { + action: "env_set" as "read", env_key: "RIGHT_KEY", _confirmed: true, pending_action_id: pendingId, + } as any); + expect((correct.details as Record).set).toBe(true); + }); + + it("retains the pending action when env.set fails transiently", async () => { + let attempts = 0; + const rpcCall = vi.fn(async (method: string) => { + if (method === "gateway.status") return statusOk(); + if (method === "env.set") { + attempts++; + if (attempts === 1) throw new Error("temporary store failure"); + return { set: true }; + } + return {}; + }); + const tool = createGatewayTool(rpcCall, mockLogger); + const gated = await tool.execute("call-p11", { + action: "env_set" as "read", env_key: "RETRY_KEY", env_value: "secret-value", + } as any); + const pendingId = (gated.details as Record).pending_action_id as string; + const confirm = { + action: "env_set" as "read", env_key: "RETRY_KEY", _confirmed: true, pending_action_id: pendingId, + } as any; + + await expect(tool.execute("call-p12", confirm)).rejects.toThrow("temporary store failure"); + const retry = await tool.execute("call-p13", confirm); + expect((retry.details as Record).set).toBe(true); }); }); diff --git a/packages/skills/src/platform-tools/tools/gateway-tool.ts b/packages/skills/src/platform-tools/tools/gateway-tool.ts index d0ca2e818e..c343f19d5e 100644 --- a/packages/skills/src/platform-tools/tools/gateway-tool.ts +++ b/packages/skills/src/platform-tools/tools/gateway-tool.ts @@ -193,13 +193,14 @@ export function stashPendingEnvSet(envKey: string, envValue: string, nowMs: numb return id; } -/** Take (and consume) a stashed env_set. One-shot: a second confirm replays nothing. */ -export function takePendingEnvSet(id: string, nowMs: number): PendingEnvSet | undefined { +export function getPendingEnvSet(id: string, nowMs: number): PendingEnvSet | undefined { sweepPendingEnvSets(nowMs); - const entry = pendingEnvSets.get(id); - if (entry === undefined) return undefined; + return pendingEnvSets.get(id); +} + +export function consumePendingEnvSet(id: string, entry: PendingEnvSet): void { + if (pendingEnvSets.get(id) !== entry) return; pendingEnvSets.delete(id); - return entry; } export function confirmationRequiredHint(action: string): string { @@ -411,8 +412,9 @@ export function createGatewayTool( // model still holds. One-shot + TTL'd; an expired/unknown id gets // an honest error rather than silently writing the mangled value. const pendingActionId = readStringParam(p, "pending_action_id", false); + let pendingEnvSet: PendingEnvSet | undefined; if (p._confirmed === true && pendingActionId !== undefined && pendingActionId.length > 0) { - const pending = takePendingEnvSet(pendingActionId, systemNowMs()); + const pending = getPendingEnvSet(pendingActionId, systemNowMs()); if (pending === undefined) { return { error: "pending_action_expired", @@ -432,6 +434,7 @@ export function createGatewayTool( }; } envValue = pending.envValue; + pendingEnvSet = pending; } if (envValue === undefined || envValue.length === 0) { return { @@ -537,6 +540,9 @@ export function createGatewayTool( }; } const result = await rpcCall("env.set", { key: envKey, value: envValue, _trustLevel }); + if (pendingActionId !== undefined && pendingEnvSet !== undefined) { + consumePendingEnvSet(pendingActionId, pendingEnvSet); + } // Return result but strip any value that might have leaked through return typeof result === "object" && result !== null ? { ...(result as Record), value: undefined } diff --git a/packages/skills/src/skills/bridge/mcp-tool-bridge.test.ts b/packages/skills/src/skills/bridge/mcp-tool-bridge.test.ts index 3d078d356f..7d43be28a7 100644 --- a/packages/skills/src/skills/bridge/mcp-tool-bridge.test.ts +++ b/packages/skills/src/skills/bridge/mcp-tool-bridge.test.ts @@ -6,6 +6,7 @@ import { ok, err } from "@comis/shared"; import { Type } from "typebox"; +import { Value } from "typebox/value"; import { describe, it, expect, vi } from "vitest"; import { runWithContext } from "@comis/core"; import type { McpToolDefinition, McpClientManager } from "../integrations/mcp-client/index.js"; @@ -173,10 +174,15 @@ describe("jsonSchemaToTypeBox", () => { expect((result as any).type).toBe("object"); }); - it("falls back to Any for unknown types", () => { + it("converts null without widening it to Any", () => { const result = jsonSchemaToTypeBox({ type: "null" }); - // typebox 1.x Any produces an empty schema {} - expect((result as any).type).toBeUndefined(); + expect((result as any).type).toBe("null"); + }); + + it("converts JSON Schema type arrays into typed unions", () => { + const result = jsonSchemaToTypeBox({ type: ["string", "null"] }) as Record; + expect(JSON.stringify(result)).toContain('"type":"string"'); + expect(JSON.stringify(result)).toContain('"type":"null"'); }); it("falls back to Any for missing type", () => { @@ -913,16 +919,6 @@ describe("mcpToolsToAgentTools - wrapExternalContent integration", () => { // Composed schemas // --------------------------------------------------------------------------- -/** - * anyOf / oneOf / allOf collapsed to Type.Any(), which erases the type entirely: - * the model is told nothing about the shape, and local validation accepts any - * value — so a wrong type travels all the way to the MCP server and comes back - * as an opaque -32602. That is the same silent-information-loss class as the - * stripped numeric bounds and enums above. - * - * Observed live: a model passed an object-typed parameter as a string, local - * validation waved it through, and the server rejected it twice. - */ describe("jsonSchemaToTypeBox preserves composed schemas", () => { it("converts anyOf into a union that still rejects a wrong type", () => { const schema = jsonSchemaToTypeBox({ @@ -930,13 +926,25 @@ describe("jsonSchemaToTypeBox preserves composed schemas", () => { }); // A union, not an untyped Any — Any has no discriminating keywords at all. expect(JSON.stringify(schema)).toMatch(/anyOf|oneOf/); + expect(JSON.stringify(schema)).toContain('"type":"null"'); + expect(Value.Check(schema, null)).toBe(true); + expect(Value.Check(schema, { a: "ok" })).toBe(true); + expect(Value.Check(schema, 42)).toBe(false); }); - it("converts oneOf into a union", () => { + it("preserves oneOf exactly and rejects overlapping matches", () => { const schema = jsonSchemaToTypeBox({ - oneOf: [{ type: "string" }, { type: "number" }], + oneOf: [ + { type: "number", minimum: 0 }, + { type: "number", maximum: 10 }, + ], }); - expect(JSON.stringify(schema)).toMatch(/anyOf|oneOf/); + const schemaRecord = schema as Record; + expect(schemaRecord.oneOf).toBeDefined(); + expect(schemaRecord.anyOf).toBeUndefined(); + expect(Value.Check(schema, -1)).toBe(true); + expect(Value.Check(schema, 20)).toBe(true); + expect(Value.Check(schema, 5)).toBe(false); }); it("keeps the object branch typed inside anyOf", () => { diff --git a/packages/skills/src/skills/bridge/mcp-tool-bridge.ts b/packages/skills/src/skills/bridge/mcp-tool-bridge.ts index cb99b90a53..6be083cd81 100644 --- a/packages/skills/src/skills/bridge/mcp-tool-bridge.ts +++ b/packages/skills/src/skills/bridge/mcp-tool-bridge.ts @@ -77,9 +77,12 @@ export function jsonSchemaToTypeBox(schema: Record): TSchema { // validation accepted any value, and a wrong type only surfaced as an opaque // MCP -32602 from the server. Live: an object-typed parameter sent as a // string, waved through locally, rejected twice upstream. - const anyOf = schema.anyOf ?? schema.oneOf; - if (Array.isArray(anyOf) && anyOf.length > 0) { - const variants = (anyOf as Array>).map(jsonSchemaToTypeBox); + if (Array.isArray(schema.oneOf) && schema.oneOf.length > 0) { + const variants = (schema.oneOf as Array>).map(jsonSchemaToTypeBox); + return Type.Unsafe({ ...annotations, oneOf: variants }); + } + if (Array.isArray(schema.anyOf) && schema.anyOf.length > 0) { + const variants = (schema.anyOf as Array>).map(jsonSchemaToTypeBox); return Type.Union(variants, annotations); } if (Array.isArray(schema.allOf) && schema.allOf.length > 0) { @@ -87,6 +90,17 @@ export function jsonSchemaToTypeBox(schema: Record): TSchema { return Type.Intersect(parts, annotations); } + if (Array.isArray(type) && type.length > 0) { + return Type.Union( + type.map((variant) => jsonSchemaToTypeBox({ ...schema, type: variant })), + annotations, + ); + } + + if (type === "null") { + return Type.Null(annotations); + } + if (type === "string") { return Type.String(annotations); } diff --git a/packages/skills/src/tools/builtin/exec-diagnostics.test.ts b/packages/skills/src/tools/builtin/exec-diagnostics.test.ts index 41b083b37d..9955f447ce 100644 --- a/packages/skills/src/tools/builtin/exec-diagnostics.test.ts +++ b/packages/skills/src/tools/builtin/exec-diagnostics.test.ts @@ -421,13 +421,12 @@ describe("matchExternallyManagedEnv (PEP 668)", () => { note: If you believe this is a mistake, please contact your Python installation provider. hint: See PEP 668 for the detailed specification.`; - it("surfaces the venv + --break-system-packages options at the head of stderr", () => { + it("surfaces a workspace virtualenv without host-package bypass guidance", () => { const hint = matchExecRecoveryHint({ stderr: PEP668, exitCode: 1, cwd: "/w" }); expect(hint).not.toBeNull(); expect(hint).toContain("PEP 668"); - expect(hint).toContain("--break-system-packages"); + expect(hint).not.toContain("--break-system-packages"); expect(hint).toContain("venv"); - // …and forbids the blind identical retry that wasted live round-trips. expect(hint).toMatch(/Do not retry the identical command/); }); diff --git a/packages/skills/src/tools/builtin/exec-diagnostics.ts b/packages/skills/src/tools/builtin/exec-diagnostics.ts index 28a4737b9d..d82502868b 100644 --- a/packages/skills/src/tools/builtin/exec-diagnostics.ts +++ b/packages/skills/src/tools/builtin/exec-diagnostics.ts @@ -194,22 +194,14 @@ const matchVenvMissing: Matcher = ({ stderr, exitCode, cwd }) => { // --------------------------------------------------------------------------- -/** - * PEP 668: a system Python refuses `pip install` with - * `error: externally-managed-environment`. Observed live: an agent burned three - * exec round-trips (bare pip → read the wall of text → retry with - * `--break-system-packages`) before its xlsx task could start. The stderr wall - * does mention the flag, but buried after a distro lecture — surface the two - * viable next steps at the HEAD so the first retry is the right one. - */ +/** PEP 668 requires dependency installation in an isolated environment. */ const matchExternallyManagedEnv: Matcher = ({ stderr, exitCode }) => { if (exitCode === 0) return null; if (!stderr || !stderr.includes("externally-managed-environment")) return null; return ( "RECOVERY HINT: This system Python is PEP 668 externally-managed — bare `pip install` " + - "will always refuse. Either install into a workspace virtualenv " + - "(python3 -m venv venv && venv/bin/pip install ) or, for a throwaway box, " + - "re-run with `pip install --break-system-packages `. Do not retry the identical command." + "will always refuse. Install into a workspace virtualenv " + + "(python3 -m venv venv && venv/bin/pip install ). Do not retry the identical command." ); }; diff --git a/test/architecture/daemon-boot-degrades-without-autonomy.test.ts b/test/architecture/daemon-boot-degrades-without-autonomy.test.ts index 3955977202..e2ee47c821 100644 --- a/test/architecture/daemon-boot-degrades-without-autonomy.test.ts +++ b/test/architecture/daemon-boot-degrades-without-autonomy.test.ts @@ -59,7 +59,7 @@ describe("the daemon boots when proactive schedulers cannot be armed", () => { // Any other missing dependency must still abort — a composition-root // regression is NOT something to boot through. const degrade = fs.readFileSync(path.join(repoRoot, "packages/daemon/src/wiring/proactive-degrade.ts"), "utf8"); - expect(degrade).toMatch(/isAutonomyDisabledProactiveMiss\(proactive\.error\)[\s\S]{0,120}throw new Error/); + expect(degrade).toMatch(/isAutonomyDisabledProactiveMiss\(proactive\.error, capEndpointUnavailableReason\)[\s\S]{0,300}throw new Error/); }); it("logs an ERROR naming what is off and the knob that turns it back on", () => { @@ -72,7 +72,7 @@ describe("the daemon boots when proactive schedulers cannot be armed", () => { // …and warns about the sub-key trap that makes this reachable by accident. expect(degradeSrc).toMatch(/autonomy\.durability/); // the daemon must actually EMIT it - expect(daemonSrc).toContain("proactiveNotArmedLogFields()"); + expect(daemonSrc).toContain("proactiveNotArmedLogFields(capEndpointUnavailableReason)"); }); it("treats the proactive surface as optional downstream (no undefined deref)", () => { From 22ae171770af2e63b8ef50be3ee63ee104db8557 Mon Sep 17 00:00:00 2001 From: comis-agent Date: Tue, 28 Jul 2026 01:50:03 +0300 Subject: [PATCH 32/33] docs: refresh incident-hardening contracts --- docs/agent-tools/messaging.mdx | 7 +++++++ docs/agent-tools/sessions.mdx | 4 ++-- docs/agents/compaction.mdx | 10 ++++++---- docs/developer-guide/event-bus.mdx | 17 +++++++++-------- .../generic-agent-architecture.md | 2 +- docs/get-started/glossary.mdx | 2 +- docs/operations/observability.mdx | 12 +++++++----- docs/reference/cli.mdx | 9 ++++++++- docs/reference/config-yaml.mdx | 2 +- docs/reference/json-rpc.mdx | 10 ++++++++-- docs/skills/mcp.mdx | 18 ++++++++++++++++++ .../bootstrap/sections/tool-descriptions.ts | 9 ++++----- .../agent/src/executor/jit-guide-injector.ts | 8 +++----- packages/agent/src/executor/prompt-compiler.ts | 11 +++-------- .../src/execution/turn-outcome-mapper.ts | 1 - 15 files changed, 78 insertions(+), 44 deletions(-) diff --git a/docs/agent-tools/messaging.mdx b/docs/agent-tools/messaging.mdx index 60cb52b7f7..6fab9604af 100644 --- a/docs/agent-tools/messaging.mdx +++ b/docs/agent-tools/messaging.mdx @@ -22,6 +22,13 @@ crossing account, thread, or conversation boundaries. The `message` tool supports **7 actions** -- `send`, `reply`, `react`, `edit`, `delete`, `fetch`, and `attach`. Comis verifies the requested coordinates against the turn's authoritative endpoint, then dispatches to its exact channel adapter instance. Not every action is supported on every platform (see the [capability matrix](#per-channel-capability-matrix) below); the tool returns a structured error if you ask for something the channel cannot do. + +The turn's final assistant text is delivered automatically. After `message` +already sends or replies with the user-facing content, the agent should finish +with `NO_REPLY`; restating the same content as the final response sends a second +message. `NO_REPLY` suppresses only that automatic final delivery. + + All actions require the `channel_type` parameter to specify which platform adapter handles the request. This is a required string identifying the channel type (e.g., `telegram`, `discord`, `slack`, `whatsapp`, `signal`, `imessage`, `line`, `irc`, `email`). ### Quick Reference diff --git a/docs/agent-tools/sessions.mdx b/docs/agent-tools/sessions.mdx index 1159bb5c8c..d203dc57e9 100644 --- a/docs/agent-tools/sessions.mdx +++ b/docs/agent-tools/sessions.mdx @@ -151,7 +151,7 @@ The four session tools (`session_search`, `session_status`, `sessions_history`, | `async` | boolean | No | Accepted for explicit asynchronous intent; every spawn runs in the background and returns `runId` immediately. | | `model` | string | No | Model override for the sub-agent | | `artifact_refs` | string[] | No | File paths for the sub-agent to reference in its workspace | - | `tool_groups` | string[] | No | Tool group names for sub-agent tool filtering (e.g., `"coding"`, `"web"`) | + | `tool_groups` | string[] | No | Tool group names for sub-agent tool filtering (e.g., `"coding"`, `"web"`). The default child profile excludes MCP tools and `message`; use `"full"` when the task genuinely requires those capabilities. | | `required_tools` | string[] | No | Tool names that must be reachable; spawn fails before execution if the child profile cannot reach one. | | `objective` | string | No | Objective statement that survives context compaction | | `include_parent_history` | `"none"` or `"summary"` | No | Include a condensed summary of the parent conversation (default: `"none"`) | @@ -160,7 +160,7 @@ The four session tools (`session_search`, `session_status`, `sessions_history`, | `max_steps` | integer | No | Maximum execution steps (floor of 30, default 50, capped at config default) | `sessions_spawn` never blocks for the child result. It returns the run ID and a background-running marker immediately; a queued spawn additionally returns `queued: true`. The result is delivered to the immutable originating conversation, so the model-facing tool cannot select an announcement channel. Use `subagents` with `action: wait` to await owned children without polling. - **When the agent reaches for it.** The tool description carries the delegation trigger, so an agent that has `sessions_spawn` on its surface is told up front to delegate rather than work inline whenever a task needs more than ~30 seconds of tool time, generates media, writes three or more files, requires deep research, or runs four or more dependent steps. Independent subtasks are spawned as multiple calls in one response and run in parallel. The fuller delegation guide (goal-oriented task phrasing, what not to delegate) is injected after the first successful spawn. + **When the agent reaches for it.** An agent that has `sessions_spawn` on its surface receives a concise delegation directive in its initial system prompt: delegate rather than work inline whenever a task needs more than ~30 seconds of tool time, generates media, writes three or more files, requires deep research, or runs four or more dependent steps. Independent subtasks are spawned as multiple calls in one response and run in parallel. The fuller guide (goal-oriented task phrasing, child tool-profile guidance, and what not to delegate) follows the first successful tool result and is delivered only once. Sub-agent results are automatically condensed (if over the token threshold) and formatted with metadata tags before being returned to the parent. See [Subagent Context Lifecycle](/agents/subagent-lifecycle) for the full pipeline. diff --git a/docs/agents/compaction.mdx b/docs/agents/compaction.mdx index bc5e24fc96..d6b20b075a 100644 --- a/docs/agents/compaction.mdx +++ b/docs/agents/compaction.mdx @@ -361,9 +361,11 @@ A three-tier escalation ensures summaries always fit within their target size: - **Truncation** -- A bounded count-only note (marker-prefixed, guaranteed to fit and to shrink below the source it replaces). -The most recent turns (the "fresh tail", default 8 turns) are always protected -from compaction, ensuring your agent has full, uncompressed access to the latest -exchanges. +The "fresh tail" starts with a configured floor of recent steps plus every +message newer than the store's persisted horizon. If residual token bounding is +needed, completed tool segments may be trimmed, but the current turn's +originating request stays pinned. See the [`freshTailTurns` config contract](/reference/config-yaml#context-engine-agents-contextengine) +for the small-window clamp and residual-token rules. One size guard applies inside the protected fresh tail: a single message whose text alone approaches the model's effective context window (an oversized tool @@ -491,7 +493,7 @@ change anything. | `contextEngine.observationDeactivationChars` | number | `80000` | Character threshold to deactivate observation masking (20K-500K) | | `contextEngine.ephemeralKeepWindow` | number | `10` | Recent ephemeral tool results to preserve from masking (1-50) | | `contextEngine.contextThreshold` | number | `0.75` | **DAG mode:** context-utilization fraction (of the turn's effective budget window — the reconciled context window under any capability-class cap) that triggers a leaf-summarization pass at the end of a turn (0.1-0.95) | -| `contextEngine.freshTailTurns` | number | `8` | **DAG mode:** most-recent steps (assistant + tool round-trips) always kept verbatim and never summarized/evicted (1-50). Reduced only on a small context window (see [config reference](/reference/config-yaml)); honoured as configured on frontier-sized windows. This is a **floor**, not a ceiling: the tail is additionally widened to cover everything the store has not yet persisted, so a turn that runs more tool round-trips than this cannot slide its own originating request out of context. Steps older than the tail are still recoverable — they resolve from the store as reconstructed history rather than verbatim blocks. | +| `contextEngine.freshTailTurns` | number | `8` | **DAG mode:** configured step floor for the verbatim tail (1-50). See the [config reference](/reference/config-yaml#context-engine-agents-contextengine) for the effective-tail, persisted-horizon, and residual-token rules. | | `contextEngine.leafChunkTokens` | number | `20000` | **DAG mode:** token cap for the oldest out-of-tail chunk summarized into one leaf summary (1K-100K); clamped at runtime to the resolved summarizer model's window — the smaller of its configured window and the probed served window when the summarizer runs on the served-bound provider — minus the summary target, template overhead, and previous-summary size, so a small compaction summarizer or a served-bound primary is never fed an over-window chunk; a single message larger than the clamped cap is replaced by a bounded deterministic extraction (no LLM call) | | `contextEngine.leafTargetTokens` | number | `1200` | **DAG mode:** target token size for a leaf summary (96-5000) | | `contextEngine.deferCompaction` | boolean | `true` | **DAG mode:** run the afterTurn leaf + condense passes in the background on the per-conversation serializer (never blocking the turn). `false` runs them inline (deterministic, for tests) | diff --git a/docs/developer-guide/event-bus.mdx b/docs/developer-guide/event-bus.mdx index e5b41fb858..2b855b33dc 100644 --- a/docs/developer-guide/event-bus.mdx +++ b/docs/developer-guide/event-bus.mdx @@ -63,11 +63,11 @@ A quick reference for the events developers are most likely to need when buildin | `plugin:registered` | Infra | Plugin loaded | Plugin lifecycle tracking | | `diagnostic:message_processed` | Infra | Full message lifecycle complete | End-to-end metrics | -## Complete Event Reference +## Event Reference - + Message lifecycle, session management, subagent lifecycle, compaction, context engine lifecycle, context engine metrics, response filtering, execution control, and dead-letter queue events. @@ -80,6 +80,7 @@ Message lifecycle, session management, subagent lifecycle, compaction, context e | `compaction:flush` | sessionKey, memoriesWritten, trigger, success, timestamp | | `context:masked` | agentId, sessionKey, maskedCount, totalChars, persistedToDisk, timestamp | | `context:compacted` | agentId, sessionKey, fallbackLevel, attempts, originalMessages, keptMessages, timestamp | +| `context:budget_computed` | agentId, sessionKey, windowTokens, rawContextWindowTokens, windowCapSource, systemTokens, freshTailTokens, budgetedHistoryTokens, keptCount, assembledInputTokens, outputHeadroom, verdict, freshTailSteps, freshTailStepsConfigured, originatingRequestRetained, freshTailTrimmedCount | | `context:rehydrated` | agentId, sessionKey, sectionsInjected, filesInjected, skillsInjected, overflowStripped, timestamp | | `context:overflow` | agentId, sessionKey, contextTokens, budgetTokens, recoveryAction, timestamp | | `context:evicted` | agentId, sessionKey, evictedCount, evictedChars, categories, timestamp | @@ -124,7 +125,7 @@ See [Compaction](/agents/compaction) for details on what triggers each event. - + Skill lifecycle, tool execution, audit logging, observability (tokens and latency), cache break detection, model failover, security, execution graph, per-node sub-agent budget breaches, pipeline authoring telemetry, small-model graph authoring (repair + intent synthesis), provider health, SEP (step-execution planning), and exec command blocking events. @@ -171,7 +172,7 @@ Skill lifecycle, tool execution, audit logging, observability (tokens and latenc | `skill:rejected` | skillName, reason, violations, timestamp | | `skill:updated` | skillName, scope, agentId, timestamp | | `skills:reloaded` | agentId, skillCount, timestamp | -| `tool:executed` | toolName, durationMs, success, timestamp, userId, traceId, agentId, sessionKey, params, errorMessage, errorKind, description, truncated, fullChars, returnedChars | +| `tool:executed` | toolName, toolCallId, durationMs, success, backgrounded, timestamp, userId, traceId, agentId, sessionKey, params, errorMessage, errorKind, description, truncated, fullChars, returnedChars | | `tool:policy_filtered` | profile, agentId, filtered, timestamp | | `tool:started` | toolName, toolCallId, timestamp, agentId, sessionKey, traceId, description | @@ -218,7 +219,7 @@ See [Resilience: Self-healing delivery retries](/agents/resilience#self-healing- - + Channel registration, sender blocking, command queuing, streaming delivery, typing indicators, auto-reply, send policies, debounce, group history, follow-up chains, priority lanes, elevated routing, retry engine, ack reactions, inbound reaction capture, steer lifecycle, block coalescing, unified delivery pipeline, delivery queue, channel health monitoring, delivery hooks, and sub-agent proxy typing events. @@ -286,8 +287,8 @@ Approval gates, config patches, plugin lifecycle, hook execution, auth token rot | `approval:resolved` | requestId, approved, approvedBy, reason, resolvedAt | | `auth:token_rotated` | provider, profileName, expiresAtMs, timestamp | | `background_task:cancelled` | agentId, taskId, toolName, timestamp | -| `background_task:completed` | agentId, taskId, toolName, durationMs, origin, timestamp | -| `background_task:failed` | agentId, taskId, toolName, error, durationMs, origin, timestamp | +| `background_task:completed` | agentId, taskId, toolName, durationMs, origin, dispatchRedelivery, toolCallId, sessionKey, traceId, timestamp | +| `background_task:failed` | agentId, taskId, toolName, error, errorKind, durationMs, origin, dispatchRedelivery, toolCallId, sessionKey, traceId, timestamp | | `background_task:notified` | agentId, taskId, toolName, sessionKey, notified, reason, traceId, timestamp, trajectoryRecorded | | `background_task:promoted` | agentId, taskId, toolName, timestamp | | `background_task:reentered` | taskId, agentId, sessionKey, hopCount, traceId, timestamp | @@ -328,7 +329,7 @@ Approval gates, config patches, plugin lifecycle, hook execution, auth token rot | `scheduler:heartbeat_wake_terminal` | correlationId, target, lane, retainedReason, status, cancellationReason, eventEntryCount, durationMs, errorKind, timestamp | | `scheduler:job_suspended` | jobId, jobName, agentId, consecutiveErrors, lastError, timestamp, deliveryTarget | | `scheduler:wake_gate` | jobId, agentId, wake, durationMs, toolCalls, estTurnsSaved, failedOpen, timestamp | -| `secret:accessed` | secretName, agentId, outcome, timestamp | +| `secret:accessed` | secretName, agentId, outcome, sessionKey, traceId, timestamp | | `secret:modified` | secretName, action, timestamp | | `security:warn` | category, agentId, message, timestamp | | `system:error` | error, source | diff --git a/docs/developer-guide/generic-agent-architecture.md b/docs/developer-guide/generic-agent-architecture.md index 712fafd59b..c7437cc0bc 100644 --- a/docs/developer-guide/generic-agent-architecture.md +++ b/docs/developer-guide/generic-agent-architecture.md @@ -46,7 +46,7 @@ Execution state such as sender trust, locale policy, selected skills, and compac `ResponseLocalePolicy` is an open, strict domain type. Explicit locale tags are canonicalized with `Intl.getCanonicalLocales`; locale is not a closed language union. Unicode script analysis may support search, token estimation, bidirectional safety, and diagnostics, but it does not coerce response language. -Translation target is separate from surrounding response locale. A validated request or operator locale enables post-generation script enforcement. When no locale metadata exists, a non-Latin current request may produce an open `und-