Locally-spawned agents never register with relaycast (send_dm agent_not_found) — briefing never delivered; root cause behind #96/#97; reproduces on two machines

[khaliqgant]

Summary

Locally-spawned implementer/reviewer agents never complete their relaycast registration. They run (the codex/claude process is alive and does work from its --task arg), but the broker/engine cannot address them, so the critical briefing DM fails and the agent proceeds un-briefed. Reproduced on two independent machines — this is not environmental to one host.

This is the root cause behind #96 (fabricated on-call policy) and the abort-not-firing in #97: both are downstream of "the briefing never reaches the agent."

The signal

On a machine where errors serialize with enumerable fields, the failure is explicit:

Relaycast publish failed: relaycast send_dm failed:
  API error (agent_not_found): Agent "ar-53-impl-relayed" not found

And the broker roster confirms it — the agent is NONE registered (or status: "unknown" with no pid/cli), while ps shows its codex process alive at ~0% CPU. So: process up, relaycast registration never completed, send_dm → agent_not_found → briefing injection exhausts its retries (max delivery retries exceeded, or the opaque [factory] error {} on published 0.1.19 where #83's fix hasn't shipped).

Reproduced on both machines

host	factory	node for the agent's agent-relay mcp	result
laptop	published 0.1.19 + local #97 build	mise node 22 (correct)	not registered → delivery fails, 4/4 dispatches
sf-mini	published 0.1.19	mise node 22 (correct)	not registered → send_dm agent_not_found

On the laptop this is the same failure I mis-attributed to the Slack gate in #94 and to #96/#97 — all of them are this. The one configuration that did brief successfully was enrolled-node placement (relayed#47: laptop orchestrator → agent placed on sf-mini's enrolled factory-node): that agent registered and followed its injected instructions. So the break is specific to the internal owned-broker spawn+register path, not the enrolled-node path.

What changed

The 3/3 verified E2E (2026-07-16, published 0.1.17, laptop, internal owned-broker) briefed fine. The same internal path now fails to register on both machines. Candidates, not yet isolated:

• a relaycast/engine-side change in how agent-relay mcp agents register (the error is a relaycast API error, agent_not_found);
• the agent's MCP server failing to start or connect (its command/args run agent-relay/dist/cli/index.js mcp under the spawn env);
• an owned-broker vs enrolled-node registration difference that a recent broker/CLI bump exposed.

Why this is the blocker

Until local-spawned agents register, no internal-backend factory dispatch can brief its agents on either machine — every run produces an un-briefed agent that guesses (#96) or stalls. It disables the entire human-steering premise and makes local E2E untrustworthy. The only working config today is enrolled-node placement, which loops back to #84/#85.

Repro

1. factory dispatch <github issue> --config <internal-backend config> (no placement).
2. agent-relay agent list → the spawned agents show status: "unknown", no pid/cli — a name only.
3. Dispatch log → send_dm ... agent_not_found (or max delivery retries exceeded, or [factory] error {} pre-[factory] error {} — raw Errors serialize to {} (message/stack are non-enumerable); describeError() already exists one line away #83).
4. Agent proceeds on the bare triage task.

Diagnostic to add

Log the agent's MCP-server startup outcome (connected / failed) at spawn, and surface agent_not_found distinctly from a generic delivery timeout. Right now the registration failure is invisible unless you cross-reference agent list against the process table.

Refs: #96, #97, #94, #84, #85, #50/#51 (earlier cold-start ack race — likely the same family).

[khaliqgant]

Correcting this issue's framing — it is NOT "agents mysteriously never register." Registration is a normal SDK call; the bug is that it either never boots or registers into the wrong workspace. Traced against AgentWorkforce/relay.

How registration actually works (confirmed in relay source)

When the spawned agent's agent-relay mcp stdio server boots, it calls the SDK thin-client agents.registerOrRotate({ name, type }) in resolveStdioBootstrapOptions (packages/cli/src/cli/agent-relay-mcp.ts:903-927), scoped to RELAY_WORKSPACE_KEY/RELAY_API_KEY at RELAY_BASE_URL. That RPC creates the workspace-scoped record that send_dm(to=name) resolves against. No other step creates it — not the WS connect, not the subprocess existing.

So agent_not_found means exactly one of:

1. The MCP server never booted, or registerOrRotate threw before the DM → no record. Boot skips registration entirely if workspaceKey or agentName is missing (agent-relay-mcp.ts:912-914); it exits(1) if the RPC rejects (:957-963).
2. The agent registered into a different workspace / base_url than factory's send_dm searches (agent-relay-mcp.ts:941, :916 scope the record to whatever key/url the agent got).

Nothing in relay's registration/MCP path changed in the 2026-07-15→17 window (last touched c3c147b34, 2026-07-13). So this is config/environment, not a relay regression — consistent with the 3/3 E2E working on 0.1.17 (2026-07-16) and failing now.

The specific suspect: codex strips MCP-config creds; factory's spawn path doesn't compensate

The relay broker's native spawn writes the codex MCP config and then injects RELAY_API_KEY directly into the process env because "codex strips API keys from .mcp.json" (crates/broker/src/snippets.rs:307-311). Factory's internal path does not use that native path — it spawns via HarnessDriverClient.spawnPty (src/fleet/internal-fleet-client.ts:167) and passes creds only through codex's --config mcp_servers.agent-relay.env={...} (codexHarnessArgs, :931).

If codex strips that inline env (the exact thing the broker compensates for), the codex MCP boots with no workspace key → resolveStdioBootstrapOptions returns early, never registers → agent_not_found. That also matches the observation that input.agentToken is never populated on this path (internal-fleet-client.ts:918), so there's no at_live_ token to fall back on either.

This is the "use the SDK, nothing out of band" gap: the placement/relay backend path (messaging.placement.spawn) spawns broker-native and registered fine (relayed#47 briefed correctly); the internal owned-broker path reconstructs the codex cred wiring itself and loses it.

What I have NOT yet isolated

Which of (1) vs (2) it is. Both are consistent with the evidence (process alive ~0% CPU, agent list shows the name at status:"unknown" no pid/cli, send_dm → agent_not_found on both laptop and sf-mini with correct node). The decisive test is one instrumented run capturing the codex child's agent-relay mcp stderr — whether the MCP booted, whether registerOrRotate ran, and into which workspace/base_url — vs the workspace factory's send_dm uses.

Corrected ask

Not "why don't agents register" but: ensure the internal spawn path delivers relay creds to the codex MCP the same way the broker-native path does (direct process-env injection, not just inline MCP env), and assert the agent registers into the same workspace/base_url the orchestrator sends from. Retitle accordingly.

(Amends my earlier comments on #94/#96/#97, which attributed this to "agents never register" — the mechanism is SDK registerOrRotate failing to run or registering off-workspace, not a missing registration concept.)

[khaliqgant]

Root cause pinned with a clean run on current main (has #90 so errors are finally visible). Retitle: it is not "agents never register" — it is that the CRITICAL TASK INJECTION into the codex implementer times out.

The real error (visible now via #90)

factory dispatch on an unanswerable issue, babysitter.enabled + slack.channel, current main build:

[factory] Slack sync soft-degraded but webhook delivery is healthy; continuing Slack writeback   ← #95 works
[factory] task injection target not registered yet; retrying
  {"to":"ar-56-impl-relayed","attempt":1,"error":"Timed out waiting for delivery_injected for http_e4d53ac…"}
[factory] error {"message":"Critical delivery failed to ar-56-impl-relayed: max delivery retries exceeded"}   ×12

The task is sent, but factory waits for a delivery_injected confirmation that never arrives, retries, and exhausts. Because the task never lands, the implementer never receives the ask instructions — so it can never emit [factory-needs-input]. The ask round trip cannot even begin. This is upstream of #95's gate fix, which works correctly here.

It is codex-specific

Only ar-56-impl-relayed (codex, spawn:codex) fails — 12× "Critical delivery failed", and the sole "task injection target not registered" line names the impl. ar-56-review-relayed (claude, spawn:claude) is not in the failures. So claude accepts the injection; codex does not.

Earlier findings corrected

• "Agents never register" (my prior framing) is wrong. The codex agent's MCP child has the right creds (RELAY_WORKSPACE_KEY=rk_live_…), the agent record IS created in the correct workspace, and with no slack config (Triage/clarification escalation should comment on the source GitHub issue, not require a Slack channel #55) the codex implementer received its task and did real work (wrote a proper file on a proper branch, no failure). So registration and basic injection work.
• The agent list status:"unknown" I leaned on is a liveness-reporting gap, not "unregistered" — the agent works regardless.
• The discriminator is slack+babysitter: fails 4/4 (relayed#48/Local dispatch: agent completes + pushes, but factory bails before auto-PR (delivery_injected ack race on cold-start agent) #50/fix(factory): resolve delivery_injected race on fresh-spawn cold start (reopens #50) #53/feat(factory): [SHADOW] escalate to GitHub issue comment when Slack channel absent (#55) #56), succeeds without slack (Triage/clarification escalation should comment on the source GitHub issue, not require a Slack channel #55). The babysitter path posts a Slack dispatch thread before #sendImplementerTask (factory.ts ~line 1334), which delays the injection; codex's injectable window is timing-sensitive and the delayed injection misses it. Triage/clarification escalation should comment on the source GitHub issue, not require a Slack channel #55 (immediate injection, no slack step) hits the window.

Where the actual fix lives

This is PTY injection into an interactive "driven" agent (codex), which AgentWorkforce/relay is actively working: #1288 fix/pty-injection-exit-race, escape-aware paced injection, ignore commands echoed during injection, let driven agents receive relay messages (Ctrl+] toggle + flush through interactive hold). The codex TUI holds interactive and factory's injection doesn't reliably break through within the retry window.

Two concrete directions:

1. Relay-side: make codex reliably consume an injected message (the interactive-hold flush / paced injection work) so delivery_injected fires.
2. Factory-side: don't let the pre-injection Slack-thread post delay the critical task injection — inject the implementer task first, establish the Slack thread after; and/or widen the injection retry window for codex.

Impact on the human-steering demo

The Slack gate (#95) is necessary but not sufficient. Until codex task injection lands reliably with babysitter+slack, the ask round trip can't be demonstrated — the implementer never receives the instructions that let it ask. The claude reviewer receiving its injection cleanly is the proof that the ask mechanism itself is fine; it's codex delivery that's blocking.

Retitle suggestion: "Critical task injection into the codex implementer times out with babysitter+slack (delivery_injected never fires) — blocks human-steering; claude injects fine".

[khaliqgant]

Tested a factory-side fix (inject implementer task before the Slack dispatch thread). It does NOT fix it — reporting the negative result so the ordering hypothesis is ruled out.

Result

Reordered factory.ts so #sendImplementerTask runs before #ensureSlackDispatchThread, rebuilt, re-ran the same babysitter+slack unanswerable issue. The codex implementer injection still fails: 5 retries of task injection target not registered yet ... Timed out waiting for delivery_injected, then a terminal Timed out waiting for delivery_injected. Reverted.

So it is not a timing/ordering problem — injecting first doesn't help. Even with the task delivered before any Slack write, codex never confirms delivery_injected.

The actual root (from the code)

Factory spawns the codex implementer with an initial task (internal-fleet-client.ts:156 task: input.task, plus spawnMode: 'task_exit', exitAfterTask: true) — codex starts working immediately — and then injects the full critical briefing on top via #sendImplementerTask. Injecting into a codex that is already working (not parked at a receptive prompt) is what fails to confirm delivery_injected.

This is racy, which is why the evidence looked config-dependent: it happened to succeed once for a small no-slack task (relayed#55), and reliably fails with the larger babysitter+slack briefing (relayed#48/#50/#53/#56/#57). The claude reviewer succeeds because claude accepts injection differently. It's the intersection of a factory design choice (inject-on-top of a spawned-with-task codex) and relay's known-hard PTY-injection-into-driven-codex problem (active work: #1288, escape-aware paced injection, interactive-hold flush).

Two real paths (neither is "reorder")

1. Relay-side (durable): make codex reliably consume an injected message through its interactive hold so delivery_injected fires — the paced-injection / Ctrl+]-flush work already in flight. Not a factory fix.
2. Factory-side (removes the fragile pattern): spawn the codex implementer with the complete task (PR flow + ask instructions folded into input.task) so there is no post-spawn critical injection to race. On main the ask instructions are already unconditional, so most of the briefing can move into the spawn task. The only post-spawn injection that must remain is the human's clarification reply — which is a small, single message delivered when codex is idle-waiting (the state where injection works), not on top of a busy boot.

Retitle stands (codex task-injection), and (2) is the concrete factory-side fix if relay's injection work isn't imminent.