Skip to content

relay backend: dispatch exits after placement, so nothing publishes the PR or parks the issue — remote work never reaches human-review #85

Description

@khaliqgant

Summary

With --backend relay, factory dispatch exits immediately after placing the agents. Everything that turns committed work into a PR parked in factory:human-review is driven by the agent-exit handler inside that process — so under placement it never runs. The remote agents do the work correctly, commit, and push; then nothing publishes the PR and the issue is stranded in factory:in-progress.

Found proving cross-node placement for AgentWorkforce/cloud#2655. Companion to #84 (without #84's fix, placement picks the orchestrator's own machine, which is why this never showed up locally).

Observed

factory dispatch /github/repos/AgentWorkforce__relayed/issues/by-id/47.json --backend relay (0.1.18 + #84's one-line fix so placement actually targets the node):

  • Both agents placed on the remote node sf-mini — verified via the spawn ack (node = sf-mini), agent-relay node agent list on the node, new codex PIDs there, and the branch existing in sf-mini's clone. Zero ar-47 processes on the orchestrator.
  • ar-47-impl-relayed did the work correctly: commit 85e27a9 "Add cross-node dev-server proof", CROSS-NODE-PROOF.md +17 lines, pushed to origin/agent/47-cross-node-proof.
  • Implementer log ends: No PR created or merged; Factory was notified/exit.
  • Result: no PR. Issue feat(factory): dispatch relayflows from integration events #47 left at factory,factory:in-progress. The dispatch process had exited ~4 minutes earlier (its stdout is just the spawn summary, 21 lines).

Cause

Two independent reasons, both structural:

  1. No exit watcher without an owned broker. cli/fleet.js only wires ownsBroker/ownedBrokerAgentExitTimeoutMs for the internal backend. createFleet({backend:'relay'}) returns RelayFleetClient, dispatch has no broker to wait on, and the CLI returns as soon as placement acks. Nothing observes the agents exiting, so factory.ts:2284's exit handler — and with it #tryPublishImplementerPr — never fires. The implementer's [factory-pr-ready]-style DM to factory is also delivered to a process that no longer exists.

  2. The PR fallback is clone-local, but under placement the clone is on the node. #tryPublishImplementerPr publishes from spec.clonePath, which is the orchestrator's path. Under relay placement the branch is created in the node's clone. Even with the orchestrator kept alive, the fallback would inspect the wrong machine's working copy (here it happens to be the same absolute path on both boxes, which masks the bug — it would break the moment a node's clone path differs).

This matters because the fallback is not an edge case. Its own comment at factory.ts:2275-2282 says agents "reliably COMMIT their work to a feature branch but often exit ... before running gh pr create — the root reason a dispatch never reached human-review even after the #67 fixes." All 3 local E2E runs on 0.1.17 reached human-review via this fallback, not the primary path. Under placement that safety net is gone, so the failure is the default rather than the exception.

Why it's a hosted blocker

Cross-node execution works. But dispatch --backend relay is effectively fire-and-forget: work happens on the node and never reaches the human-review gate. "Hosted" needs the loop to survive placement, which means either:

  • the orchestrator runs as a daemon (factory start) that stays alive to service exits/writeback — then one-shot dispatch --backend relay should say so rather than silently no-op the back half; or
  • the PR-publish + parking becomes node-side or durable (the node has the clone and the branch; it is the natural publisher), so the loop does not depend on an orchestrator process staying up.

Given #2 above, the second is likely the right shape for cloud#2655 — the clone-local assumption in #tryPublishImplementerPr is orchestrator-centric and does not survive nodes as cattle.

Repro

  1. Enroll a fleet node advertising spawn:codex/spawn:claude with a clonePaths entry for the repo.
  2. Apply dispatch drops spec.repo when calling fleet.spawn — relay placement can't filter by repo, silently runs on the orchestrator's own node #84's fix (else placement lands on the orchestrator and the bug is invisible).
  3. factory dispatch <mount path> --backend relay on a labeled issue.
  4. Observe: branch + commit pushed from the node; dispatch exited minutes earlier; no PR; issue stuck at factory:in-progress.

Refs: AgentWorkforce/cloud#2655, AgentWorkforce/cloud#2656, #84, #67, #69, #71.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions