fix: the skill shipped to agents was the stalest surface of all - #224
Merged
Conversation
skills/clawrouter/SKILL.md is loaded into an agent's context to tell it what this router does. Nobody had swept it, and it showed: save 67% a sixth savings figure 55+ models twice — the catalog serves 66 a routing table describing a configuration that no longer exists That table quoted Gemini Flash, DeepSeek Chat, Claude Opus and o3 as the four tier primaries. config.ts routes to gemini-2.5-flash, kimi-k2.7, gemini-3.1-pro and grok-4-1-fast-reasoning. It also priced Opus at $75/M, which is wrong by any reading — Opus 5 is $5/$25. Rewritten from config.ts and the live catalog. The per-tier "% of traffic" and "% savings" numbers are dropped rather than restated: they were measured against a baseline nobody wrote down, and the published figure is blended across a stated workload mix. Removing an unverifiable claim beats replacing it with a different unverifiable claim. Frontmatter takes literals, not markers — it is a YAML string an agent reads, so a comment would be part of it. Asserted in brand-numbers.test.ts alongside package.json, which has the same constraint. README's ecosystem card said 50+ models; bound. Left alone: "LLM Router Benchmark: 46 Models" is the title of a benchmark that tested 46 models. Restating it would misrepresent what was run.
📝 WalkthroughWalkthroughUpdated ClawRouter marketing text with new savings, model-count, payment, and tier-pricing details. Added a Vitest check that validates skill frontmatter figures against ChangesBrand number synchronization
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 inconclusive)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/router/brand-numbers.test.ts`:
- Around line 143-155: Validate that the closing "\n---" delimiter was found
before slicing frontmatter in the shared test setup. Add an explicit assertion
on the delimiter index, then preserve the existing frontmatter extraction and
savings/model assertions.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 91fcb8bb-8a57-4de5-aaa7-9355da7fdbb4
📒 Files selected for processing (3)
README.mdskills/clawrouter/SKILL.mdsrc/router/brand-numbers.test.ts
Comment on lines
+143
to
+155
| const skill = readFileSync("skills/clawrouter/SKILL.md", "utf8"); | ||
| const frontmatter = skill.slice(0, skill.indexOf("\n---", 4)); | ||
| const n = JSON.parse(readFileSync("brand-numbers.json", "utf8")) as { | ||
| models: { chatVisible: number }; | ||
| savings: { autoVsBaselinePct: number }; | ||
| }; | ||
|
|
||
| it("quotes the published savings figure", () => { | ||
| expect(frontmatter).toContain(`save ${n.savings.autoVsBaselinePct}% on inference costs`); | ||
| }); | ||
|
|
||
| it("quotes the published model count", () => { | ||
| expect(frontmatter).toContain(`${n.models.chatVisible} models`); |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Validate the frontmatter boundary before asserting values.
If the closing \n--- delimiter is missing, indexOf returns -1, so slice(0, -1) searches almost the entire markdown file. Matching text in the body could then make this test pass without valid frontmatter.
Add an explicit delimiter assertion before slicing.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/router/brand-numbers.test.ts` around lines 143 - 155, Validate that the
closing "\n---" delimiter was found before slicing frontmatter in the shared
test setup. Add an explicit assertion on the delimiter index, then preserve the
existing frontmatter extraction and savings/model assertions.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found by sweeping for the shape of a claim — a number in front of the claim vocabulary — rather than for phrasings anyone remembered. Three earlier passes over these repos had each grepped differently and each missed a different subset.
What was stale
See the commit for the per-file list. All values now come from the published artifact, and
sync-brand-numbers --checkis clean outsideCHANGELOG.md, which is history and stays as written.Frontmatter takes literals, not markers
A
SKILL.mdYAMLdescription:is a string an agent reads when deciding whether to load the skill — an HTML comment there is part of the string, not invisible. Same for headings, where it leaks into the anchor slug. Those counts are asserted in the test suite instead, the same trade already made forpackage.jsonand the shields badges.Summary by CodeRabbit
Documentation
Tests