Skip to content

Fix stale GitHub dispatch recovery and diagnostics#119

Merged
khaliqgant merged 8 commits into
mainfrom
agent/fix-stale-github-dispatch-telemetry
Jul 19, 2026
Merged

Fix stale GitHub dispatch recovery and diagnostics#119
khaliqgant merged 8 commits into
mainfrom
agent/fix-stale-github-dispatch-telemetry

Conversation

@khaliqgant

@khaliqgant khaliqgant commented Jul 19, 2026

Copy link
Copy Markdown
Member

Root cause

A persisted GitHub dispatch could be resumed without re-reading the live issue. HoopSheet issue #41 was already closed, but its stale durable lifecycle and by-id alias still allowed Factory to spawn the reviewer. Reviewer spawns also inherited the broker restart default, so teardown could re-register the name as an unowned orphan. Cancellation telemetry then collapsed the result into a generic dispatch_failed/abandoned event.

Changes

  • revalidate durable GitHub dispatches before spawning and abandon only confirmed terminal source state
  • prefer canonical GitHub issue metadata over stale by-id aliases
  • keep transiently unreadable issue sources retryable
  • set reviewer restartPolicy.maxRestarts to 0
  • report structured cancellation reasons, including source_state_changed, without provider error text
  • report a privacy-safe Factory instance name from reporting.instanceName or a sole configured repo name

Validation

  • npm test -- --silent --reporter=dot: 44 files, 943 tests passed
  • npm run build: passed
  • git diff origin/main...HEAD --check: passed

Summary by cubic

Fixes stale GitHub dispatch recovery and telemetry. We now refuse to dispatch or resume closed issues, stop the broker from re-registering torn-down reviewers, and report clear, privacy-safe cancellation reasons and instance identity.

  • Bug Fixes

    • Revalidate live GitHub issues before dispatch/resume; abandon when the source is closed or not ready.
    • Prefer canonical GitHub issue metadata over stale by-id aliases; cache preferred paths and index aliases.
    • Keep transiently unreadable sources retryable (do not abandon).
    • Set reviewer restartPolicy.maxRestarts: 0 to prevent broker orphan re-registration.
    • Treat closed GitHub issues as not dispatchable.
  • Observability

    • Add structured cancellation reasons (source_state_changed, agent_spawn_failed, agent_delivery_failed, dispatch_failed) via FACTORY_CLOUD_CANCELLATION_REASONS_V1.
    • Map raw release/cancellation messages to closed categories (e.g., worker_exited -> exited, live dispatch state changed -> source_state_changed) without leaking provider error text.
    • Report a privacy-safe instance name from reporting.instanceName or the sole configured repo; trimmed, validated, and sent as instance.metadata.name.

Written for commit 02897ff. Summary will update on new commits.

Review in cubic

kjgbot and others added 6 commits July 19, 2026 13:14
The observed dispatch failure sequence — issue admitted, implementer
worker_ready, reviewer agent_spawned but never ready, Factory tears down
the workers, the broker retries an orphan, and the dashboard only shows
dispatch_failed — has a concrete cause in defaultRestartPolicy.

Factory owns durable resume/respawn for its dispatch agents and neutralizes
the broker's default restart policy with { maxRestarts: 0 } so a broker-level
retry can't re-register a name before Factory resumes it (relay#1116-family).
That opt-out was applied to implementer and babysitter but NOT reviewer, even
though routeReviewerSpec sets no restartPolicy and the reviewer shares the
implementer's dispatch lifecycle (spawned in the same batch, torn down by the
same dispatch-failure/#releaseAndTerminateAgents paths, resumed via the same
#resumeDurableDispatch flow). So a torn-down reviewer fell through to the
broker default, which re-registered its name as an orphan while the dashboard
only reported dispatch_failed.

Extend the maxRestarts:0 opt-out to the reviewer role. Add a regression test
asserting both the implementer and reviewer dispatch spawns carry
{ maxRestarts: 0 } (the reviewer's restartPolicy was previously undefined).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jul 19, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

Changes

Factory lifecycle and observability

Layer / File(s) Summary
Instance identity and cloud event contracts
src/config/schema.ts, src/config/schema.test.ts, src/observability/instance-identity.*, src/observability/events.*, src/cli/fleet.ts, src/index.ts
Adds validated factory instance names, preserves repository names, publishes instance metadata, and extends cloud event cancellation and release-reason contracts.
Live dispatch validation and cancellation reporting
src/orchestrator/factory.ts, src/orchestrator/factory.test.ts
Revalidates issue readiness, classifies spawn and delivery failures, persists lifecycle outcomes, reports cancellation reasons, and applies no-restart policies.
GitHub path selection and durable resume revalidation
src/orchestrator/factory.ts, src/orchestrator/factory.test.ts
Prefers canonical GitHub issue paths, rejects closed issues, and abandons non-resumable durable resumes with agent cleanup and lifecycle updates.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant FactoryLoop
  participant GithubIssueSource
  participant FleetClient
  participant LifecycleStore
  participant FactoryEventReporter
  FactoryLoop->>GithubIssueSource: reload issue state
  GithubIssueSource-->>FactoryLoop: current GitHub issue
  FactoryLoop->>FleetClient: spawn or release agents
  FactoryLoop->>LifecycleStore: persist lifecycle outcome
  FactoryLoop->>FactoryEventReporter: emit cancellation telemetry
Loading

Possibly related PRs

Suggested reviewers: kjgbot

Poem

A rabbit checks the issue twice,
Then tags the clouds with names precise.
Spawn failed? “Cancelled,” says the hare,
No secret error text floats there.
Canonical paths guide the way—
Durable hops resume today.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title accurately summarizes the main change: fixing stale GitHub dispatch recovery and related diagnostics.
Description check ✅ Passed The description is directly related to the changeset and matches the implemented fixes and telemetry updates.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch agent/fix-stale-github-dispatch-telemetry

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for configuring and validating explicit reporting instance names, adds structured telemetry for run cancellations, and prevents orphan reviewer agents by disabling broker-level restarts. It also ensures that durable dispatches are not resumed or dispatched if the live GitHub issue is closed or no longer ready. The review feedback highlights two key areas for improvement: first, a potential performance bottleneck where the entire directory tree is scanned to resolve preferred issue paths, and second, a state-handling bug in #abandonDurableResume where failure handoffs are not cleared from the state store after a successful worktree teardown.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread src/orchestrator/factory.ts Outdated
Comment thread src/orchestrator/factory.ts
@kjgbot
kjgbot marked this pull request as ready for review July 19, 2026 12:13
@khaliqgant
khaliqgant merged commit f2c6f9f into main Jul 19, 2026
3 checks passed
@khaliqgant
khaliqgant deleted the agent/fix-stale-github-dispatch-telemetry branch July 19, 2026 12:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants