Skip to content

fix(agent): preserve tool failure truth during locale repair - #365

Merged
anconina merged 5 commits into
mainfrom
fix/fleet-report-delivery-integrity
Jul 28, 2026
Merged

fix(agent): preserve tool failure truth during locale repair#365
anconina merged 5 commits into
mainfrom
fix/fleet-report-delivery-integrity

Conversation

@anconina

Copy link
Copy Markdown
Contributor

Intent

Investigate the 28 July 2026 Telegram fleet-report incident end to end, identify every root cause across logs, current code, deployment drift, the Ituran integration, and operator workspace policy; fix all demonstrated generic runtime defects without adding fleet-specific assumptions to the engine; validate the changes; deploy the resulting Comis build to comis-moshe; update the deployment's integration/timeouts and workspace policy where the evidence requires it; restart safely; and leave Moshe a truthful, attachment-verifiable retest path. Preserve Result/error/security/logging contracts, RED-to-GREEN history, and do not add Co-Authored-By trailers.

What Changed

  • Track tool invocation order and privacy-preserving message action/route/target identities so only later, matching successes count as recovery.
  • Preserve the original response and recorded locale-quality finding when an unrecovered tool failure makes locale repair unsafe.
  • Reuse the recovery classifier for user-facing failure notices and execution observability, with updated documentation and regression coverage.

Risk Assessment

✅ Low: The revised change is well-bounded: it carries content-free route identities at the bridge, enforces ordered exact-operation recovery through one shared classifier, and preserves locale diagnostics without reintroducing the incident path.

Testing

Diff and intent inspection, focused agent regressions, prompt-runner egress integration, and a three-scenario product transcript all passed. Initial Vitest-filter and CJS/ESM harness setup errors were corrected without source changes. Live Telegram deployment was not exercised because deployment is outside this assigned local-test phase.

Evidence: Fleet-report delivery truthfulness transcript

Three delivery scenarios passed: unrelated send and different-target attachment success preserved the truthful failure reply; exact attachment retry permitted locale repair. SHA-256: ca869161963aea464bbcf33bf16a110ba1ce79319e28bb9cf9085a100a6fa9cb

{
  "surface": "final response locale egress",
  "policy": "Arabic response enforcement",
  "result": "PASS",
  "scenarios": [
    {
      "scenario": "failed attachment followed by successful plain message",
      "tool_trace": [
        {
          "action": "attach",
          "outcome": "failure",
          "invocation_sequence": 0
        },
        {
          "action": "send",
          "outcome": "success",
          "invocation_sequence": 1
        }
      ],
      "user_visible_response": "I could not attach the fleet report because Telegram delivery failed.",
      "locale_repair_model_calls": 0,
      "locale_repair_skipped_for_unrecovered_failure": true,
      "recovery_identity_contains_raw_channel_or_attachment": false,
      "verdict": "PASS"
    },
    {
      "scenario": "failed attachment followed by successful retry of the same attachment",
      "tool_trace": [
        {
          "action": "attach",
          "outcome": "failure",
          "invocation_sequence": 0
        },
        {
          "action": "attach",
          "outcome": "success",
          "invocation_sequence": 1
        }
      ],
      "user_visible_response": "تم إرفاق تقرير الأسطول بعد نجاح إعادة محاولة التسليم.",
      "locale_repair_model_calls": 1,
      "locale_repair_skipped_for_unrecovered_failure": false,
      "recovery_identity_contains_raw_channel_or_attachment": false,
      "verdict": "PASS"
    },
    {
      "scenario": "failed attachment followed by success for a different attachment target",
      "tool_trace": [
        {
          "action": "attach",
          "outcome": "failure",
          "invocation_sequence": 0
        },
        {
          "action": "attach",
          "outcome": "success",
          "invocation_sequence": 1
        }
      ],
      "user_visible_response": "I could not attach the fleet report because Telegram delivery failed.",
      "locale_repair_model_calls": 0,
      "locale_repair_skipped_for_unrecovered_failure": true,
      "recovery_identity_contains_raw_channel_or_attachment": false,
      "verdict": "PASS"
    }
  ]
}

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 2 issues found → auto-fixed ✅
  • 🚨 packages/agent/src/executor/prompt-runner/response-locale-enforcement.ts:234 - The intent requires “fix all demonstrated generic runtime defects” and a “truthful, attachment-verifiable retest path,” but recovery is collapsed to tool name. Because message multiplexes send/reply/attach and targets, a failed report attachment followed by any successful message call—or even success before failure—makes unrecoveredToolCount zero and re-enables the truth-reversing repair. Choose either conservative blocking after any tool failure or invocation/action/route-level recovery evidence at the bridge and one authoritative shared classifier; this name-only gate is not durable.
  • ⚠️ packages/agent/src/executor/prompt-runner/response-locale-enforcement.ts:241 - The failure gate runs before locale evaluation. Since prompt assembly always supplies a policy, failed-tool turns log “repair skipped” even when enforcement is disabled; for a real enforced mismatch, the documented content-free localeQualityFinding is never recorded. Evaluate first, return silently when there is no mismatch, and record the finding before skipping model repair.

🔧 Fix: fix route-aware tool recovery during locale enforcement
✅ Re-checked - no issues remain.

✅ **Test** - passed

✅ No issues found.

  • git status --short --branch; inspected af35aa9e..6d7ab7be diff and commit history
  • Reviewed docs/developer-guide/generic-agent-architecture.md against the runtime change
  • pnpm exec vitest run --project packages/agent … (setup retry: invalid project filter)
  • pnpm --dir packages/agent exec vitest run src/bridge/tool-failure-recovery.test.ts src/bridge/pi-event-bridge.test.ts src/executor/executor-post-execution.test.ts src/executor/prompt-runner/response-locale-enforcement.test.ts
  • pnpm --dir packages/agent exec vitest run src/executor/prompt-runner/output-escalation.test.ts
  • Ran an inline Node/TypeScript harness importing applyResponseLocaleEnforcement and buildToolRecoveryIdentity across three delivery scenarios; two initial CJS/ESM harness setup attempts were corrected
  • Parsed and validated the generated evidence JSON, then recorded its SHA-256 digest
  • Inspected RED/GREEN commit ordering and checked target commit messages for forbidden Co-Authored-By: trailers
  • git status --short --untracked-files=all confirmed the worktree remained clean
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

@github-actions

Copy link
Copy Markdown

PR description incomplete

Please fill in all required sections before this PR can be reviewed.

Required sections: Description, Related Issue, Type of Change, Checklist, RED Test Proof.

For code changes in packages/*/src/**, paste the failing test output (test name + assertion error) from before the production patch in the RED Test Proof section, or write EXEMPT: <reason> for docs/CI/config-only PRs.

See CONTRIBUTING.md for the full contribution bar.

@mintlify

mintlify Bot commented Jul 28, 2026

Copy link
Copy Markdown

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
comis 🟢 Ready View Preview Jul 28, 2026, 9:45 AM

💡 Tip: Enable Workflows to automatically generate PRs for you.

@anconina
anconina merged commit 34a6e62 into main Jul 28, 2026
11 of 13 checks passed
@anconina
anconina deleted the fix/fleet-report-delivery-integrity branch July 28, 2026 10:56
@mintlify

mintlify Bot commented Jul 28, 2026

Copy link
Copy Markdown

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
comis 🟡 Building Jul 28, 2026, 9:44 AM

💡 Tip: Enable Workflows to automatically generate PRs for you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant