refactor(dream): remove learning pipeline + per-task skills with dynamic model loading

[dean0x]

Summary

This PR removes the Devflow self-learning pipeline end-to-end and restructures the Dream subsystem from a single sequential agent into N per-task agents with clean context and a hardcoded task→model map.

Delivered in two commits to allow atomic review of each concern:

• Phase A (cf853e5) — Learning pipeline removal end-to-end
• Phase B (98ffa97) — Per-task skills + dynamic model loading

---

Changes

Phase A: Learning Pipeline Removal

What was removed:

• scripts/hooks/eval-learning and eval-reinforce — SessionEnd learning batch accumulator and artifact reinforcement
• src/cli/commands/learn.ts — full devflow learn CLI command (5 subcommands)
• src/cli/hud/learning-counts.ts + components/learning-counts.ts — HUD learning counts display
• tests/learn.test.ts, tests/learning/hud-counts.test.ts, tests/learning/capacity-thresholds.test.ts, tests/learning/migration.test.ts — all learning-specific test files

What was updated:

• DreamConfig interface — now {memory, decisions, knowledge} (no learning field); coerceConfig silently drops legacy learning key
• scripts/hooks/dream-collect-tasks — learning) case is unconditional rm (delete orphaned markers on sight, not just when disabled)
• scripts/hooks/session-start-context — removed Section 1.75 (LEARNED BEHAVIORS) and all _SC2_LEARN_EN / _LEARNING_DREAM references
• src/cli/utils/migrations.ts — added purge-learning-pipeline-v1 (per-project) and purge-learning-global-v1 (global) migrations: clean up eval-learning, eval-reinforce installed hooks, orphaned learning log + config + state files, and the learning key from dream configs

Kept intentionally:

• src/cli/utils/learning-cleanup.ts — cleanSelfLearningArtifacts is called by the per-project migration
• src/cli/utils/observations.ts + observation-io.ts — decisions pipeline uses shared LearningObservation type and readObservations

Phase B: Per-Task Skills + Dynamic Model Loading

New skills (shared/skills/dream-{memory,decisions,knowledge,curation}/SKILL.md):

• Each carries the verbatim task procedure lifted from the former monolithic dream.md body — logic is byte-for-byte identical, only hosting changed (applies ADR-008: LLM judgment in skills, plumbing in scripts)
• allowed-tools: Read, Bash, Write, Edit, Glob, Grep — deliberate exception to the read-only skill default; these skills materialize artifacts (same posture as quality-gates and git)
• Lock hardening (reliability rule, AC-C6/AC-P4): replaced the give-up-fast mkdir || sleep 1 || exit 1 pattern in both the decisions skill (.observations.lock) and curation skill (.decisions.lock) with a bounded retry+backoff loop: 9 attempts, exponential backoff doubling from 1s capped at 8s, ~47s total. On exhaustion the task fails cleanly (leaves .processing for dream-recover) — never silently drops a write. No unbounded loops.
• Curation skill: acquires .decisions.lock EXACTLY ONCE across its read-modify-write; never calls decisions-append while holding it (avoids the deadlock documented in KNOWLEDGE.md)

Rewritten Dream agent (shared/agents/dream.md):

• Body is now plumbing-only: claim markers, heartbeat, multi-marker merge, then load the matching per-task skill via the Skill tool and follow it
• Supports the combined "decisions then curation" sequential spawn (Opus handles both, run decisions fully first, then curation)
• Adds four new dream skills to skills: frontmatter alongside existing devflow:apply-decisions and devflow:apply-feature-knowledge

Rewritten SessionStart spawn directive (scripts/hooks/session-start-context Section 2):

• Replaces single Agent(subagent_type="Dream") with per-task Agent() calls using a hardcoded task→model map: memory=haiku, knowledge=sonnet, decisions=opus, curation=opus
• decisions + curation co-pending → exactly ONE opus spawn whose prompt instructs running decisions skill then curation skill sequentially (prevents concurrent lock contention on .decisions.lock)
• Unknown task types are silently skipped (belt-and-suspenders — dream-collect-tasks should never emit them)
• bash -n clean; set -e intentionally absent (existing no-abort discipline, applies ADR-009/PF-008)

Registration:

• All four skills added to plugins/devflow-core-skills/.claude-plugin/plugin.json and src/cli/plugins.ts (core-skills is the correct home — skills install universally regardless of plugin selection)
• Bare names added to LEGACY_SKILL_NAMES for future cleanup migrations

---

Breaking Changes

None for end users. The devflow learn command is removed; users who call it will get a "command not found" error. The two new migrations clean up all installed artifacts automatically on devflow init.

---

Token Cost Characterization (AC-P1)

The new design spawns N per-task agents (up to 3: haiku for memory, sonnet for knowledge, opus for decisions+curation) instead of one sequential agent handling all tasks in a single context. Expected higher per-cycle token cost, justified by:

• Clean context per task: each agent starts fresh, reducing irrelevant context overhead and improving LLM focus
• Per-task model fit: memory (cheap haiku), knowledge refresh (capable sonnet), decisions/curation detection (high-quality opus) — the previous design ran everything at sonnet regardless
• Decisions quality: running decisions on Opus every deep session improves ADR/PF detection fidelity; user-accepted trade-off
• The concurrent execution model means wall-clock time is shorter even if raw token count increases

---

Reviewer Focus Areas

• shared/skills/dream-decisions/SKILL.md — bounded retry loop for .observations.lock; verify cap is 9 attempts, backoff doubles 1→2→4→8 (capped), no unbounded paths
• shared/skills/dream-curation/SKILL.md — verify lock acquired once, Edit calls happen between acquire/release, decisions-append not called while holding the lock
• scripts/hooks/session-start-context lines 183–245 — verify decisions+curation branch emits exactly ONE opus spawn; verify memory/knowledge branches emit their correct models; verify unknown types fall through to the "no known types" log path
• shared/agents/dream.md — confirm no learning remnants; confirm the "decisions then curation" combined prompt is unambiguous

---

Follow-up: 3 deferred tech-debt items resolved

Resolves the 3 items deferred during /code-review + /resolve on this branch.

• perf(hooks): single-pass dream-collect-tasks — restructured into one first-pass scan; get_mtime (stat subprocess) and basename are no longer spawned per-marker in the common case. get_mtime is now invoked only when the marker count exceeds the cap (50). Also moved the learning.* orphan sweep and disabled-feature deletion into the unconditional first pass, closing a latent gap where markers past position 50 were never swept. curation/unknown types always pass through. Signature and _DREAM_TASKS contract unchanged.
• refactor(dream): rename .reinforce.lock → .observations.lock — "reinforce" was learning-era vocabulary; the lock serializes merge-observation writes.
• refactor(dream): drop transitional sidecar/config.json fallback (clean break) — removed the runtime fallback from 6 hooks and dream-config.ts readConfig; the rename-sidecar-to-dream-v1 migration already handles upgrades (ADR-001 clean-break).

All quality gates green: npm run build, tsc --noEmit, full vitest (1475 tests), bash -n on all edited hooks.

[dean0x]

PR #238 Code Review: Inline Findings Summary

Merge Recommendation: APPROVED_WITH_CONDITIONS — Code quality is strong, but 3 BLOCKING documentation issues must be resolved before merge.

---

Blocking Issues (Must Fix Before Merge)

1. Stale skill count in CLAUDE.md (Line 75)

File: CLAUDE.md:75 | Confidence: 95%

├── shared/skills/          # 42 skills (single source of truth)

This PR adds 4 new dream-* skills, making the actual total 45. The count must be updated to reflect the new skills being added in this PR.

Fix: Change 42 skills → 45 skills

---

2. Stale skill count in README.md (Line 52)

File: README.md:52 | Confidence: 82%

**41 skills grounded in expert material.**

The literal count now disagrees with CLAUDE.md after this PR. Recommend clarifying the intent — are these 41 expert-backed skills (keeping expert framing) or 45 total skills?

Fix: Either clarify "41 expert-backed skills" or reconcile to 45.

---

3. Four new dream-* skills missing mandatory Iron Law

Files:

• shared/skills/dream-memory/SKILL.md
• shared/skills/dream-decisions/SKILL.md
• shared/skills/dream-knowledge/SKILL.md
• shared/skills/dream-curation/SKILL.md

Confidence: 90%

CLAUDE.md explicitly requires: "Each skill has one non-negotiable Iron Law in its SKILL.md." All 41 pre-existing skills have this section. These 4 new skills are the only ones without it.

Fix: Add ## Iron Law sections to each, e.g.:

• dream-memory: "Author from the queue, never fabricate — synthesize only captured turns."
• dream-decisions: "Single marker claim per session — never hand-edit numbering."
• dream-knowledge: "Refresh from current code, never from memory."
• dream-curation: "Append-only — flip status, never remove entries."

---

Should-Fix Issues (High Priority, Non-Blocking)

4. Stale orchestrator comment in dream-evaluate (Line 7)

File: scripts/hooks/dream-evaluate:7 | Confidence: 96%

# Orchestrator: sources eval-helpers + 4 feature modules after shared setup.

After removing eval-learning and eval-reinforce, the orchestrator now sources exactly 3 feature modules (eval-decisions, eval-knowledge, eval-curation).

Fix: Change 4 feature modules → 3 feature modules

---

5. Temp file leak on failed rename in migrations.ts (Lines 795–797)

File: src/cli/utils/migrations.ts:795–797 | Confidence: 82%

const tmp = configPath + '.tmp.' + process.pid;
await fs.writeFile(tmp, JSON.stringify(config, null, 2) + '\n', { encoding: 'utf-8', mode: 0o600 });
await fs.rename(tmp, configPath);

If fs.rename() throws (e.g., EACCES, EXDEV), the temp file is left behind and accumulates on disk.

Fix: Wrap in try/catch to clean up on failure:

const tmp = configPath + '.tmp.' + process.pid;
await fs.writeFile(tmp, JSON.stringify(config, null, 2) + '\n', { encoding: 'utf-8', mode: 0o600 });
try {
  await fs.rename(tmp, configPath);
} catch (err) {
  await fs.rm(tmp, { force: true }).catch(() => {});
  throw err;
}

---

6. Orphaned documentation files after learning pipeline removal

Files: docs/self-learning.md (whole file) | Confidence: 88%

The learning pipeline (eval-learning, devflow learn command) has been fully removed, but docs/self-learning.md still describes it and is unlinked from README.

Fix: git rm docs/self-learning.md and remove any inbound references from docs/working-memory.md.

---

7. Stale CLI documentation in docs/cli-reference.md

File: docs/cli-reference.md:24, 82–88 | Confidence: 97%

The entire devflow learn subcommand and --learn init flag are removed in this PR, but the CLI reference still documents them.

Fix: Delete the --learn/--no-learn init flag row (line 24) and the learn command block (lines 82–88).

---

Summary

Category	Count	Issues
Blocking	3	CLAUDE.md skill count, README.md skill count, missing Iron Laws (4 skills)
Should-Fix	4	dream-evaluate comment, temp-file leak, orphaned docs (2 files)

Pre-existing issues flagged for context (not blocking, unchanged lines):

• ensure-devflow-init still creates .devflow/learning/ — contradicts purge-learning-pipeline-v1 migration (unchanged from main, informational per review methodology)
• .reinforce.lock naming is now semantic orphan (naming debt, not behavioral issue, pre-existing)
• init.ts:1070 comment references deleted devflow learn --run-background

---

All changes outside blocking/should-fix categories are solid. The architecture refactor is clean, test coverage is excellent, and the learning pipeline removal is complete with proper migrations.

Authored by Claude Code