fix: expose per-model thinking tiers and forward reasoning_effort - #28
fix: expose per-model thinking tiers and forward reasoning_effort#28m1yuan wants to merge 1 commit into
Conversation
f1bd1b5 to
1e0f6a6
Compare
Previously all Command Code models registered reasoning:true with no thinkingLevelMap, so pi/OMP hid the xhigh/max tiers and clamped them to high. Now each model registers a thinkingLevelMap built from its supported efforts (extracted from the official command-code CLI 1.7.0 catalog into MODEL_EFFORTS), exposing exactly the tiers each model supports — e.g. deepseek-v4-flash shows off/high/max, Claude Sonnet 5 shows low..max. The selected thinking level is forwarded to /alpha/generate as params.reasoning_effort, matching the official CLI. The off level and unsupported tiers (mapped to null) are omitted from the request. Also resolve COMMANDCODE_MODELS_URL inside the entry function so the override is honored at call time instead of fixed at module load. Adds tests for per-model tier visibility, reasoning_effort passthrough, off/null omission, and the unknown-model fallback.
1e0f6a6 to
d36aac7
Compare
patlux
left a comment
There was a problem hiding this comment.
Thanks for tackling this — the underlying pi issue is real and forwarding supported reasoning_effort values is the right direction. I found three blockers that should be addressed before merge:
- Keep
MODEL_EFFORTSexactly aligned with the authoritativecommand-code@1.7.0effort catalog. - Do not infer a per-model default that upstream does not publish.
- Emit canonical OMP thinking metadata (
mode: "effort"plusefforts) separately from pi's legacythinkingLevelMap.
I verified that command-code@1.7.0 is currently npm's latest release. Its generated model catalog says models without an effort entry decide their own reasoning depth.
Non-blocking follow-ups:
- Use
mkdtemp()/tmpdir()instead of the shared hard-coded/tmp/commandcode-models-thinking-levels-test.jsonpath, and clean it up infinally. - Please add
@m1yuanto the Unreleased contributors section. - A regression fixture/test containing the exact 1.7.0 effort catalog would help prevent unsupported entries from being introduced again.
| "moonshotai/Kimi-K2.5": ["high", "max"], | ||
| "moonshotai/Kimi-K2.6": ["high", "max"], | ||
| "sakana/fugu-ultra": ["high", "xhigh"], | ||
| "tencent/hy3-paid": ["low", "medium", "high"], |
There was a problem hiding this comment.
The checked-in table does not match the authoritative command-code@1.7.0 catalog. These five entries have no adjustable effort surface in the published CLI: claude-haiku-4-5-20251001, moonshotai/Kimi-K2.5, moonshotai/Kimi-K2.6, meta/muse-spark-1.1, and tencent/hy3-paid. The CLI's generated reference/models.md marks their Efforts column as — ("models without an effort column entry decide their own reasoning depth"). Please remove these entries; otherwise the provider exposes and sends effort values that the official CLI does not support. The remaining 22 entries match the 1.7.0 effort map.
| export type ThinkingMetadata = { | ||
| thinkingLevelMap: Record<string, string | null> | ||
| thinking: { | ||
| effortMap: Record<string, string | null> |
There was a problem hiding this comment.
OMP 17.2.x expects canonical ThinkingConfig: { mode: "effort", efforts: [...], defaultLevel?, effortMap? }. mode is required, while this effortMap also contains legacy pi-only off/null values that do not belong in OMP's string-valued map. Please keep the legacy thinkingLevelMap for pi, but emit OMP metadata separately, e.g. thinking: { mode: "effort", efforts: [...efforts] }. An OMP identity map is unnecessary because omitted mappings already pass the supported effort through unchanged.
| thinking: { | ||
| effortMap, | ||
| efforts: [...efforts], | ||
| defaultLevel: efforts[efforts.length - 2] ?? efforts[0], |
There was a problem hiding this comment.
Please do not infer defaultLevel from array position. OMP actively applies this value on initial model selection and every model switch, so this makes Claude/GPT-5.6 default to xhigh, Gemini/Grok to medium, etc. The Command Code catalog publishes supported efforts but no per-model default, so this changes runtime behavior and potentially cost without an upstream source. Omit defaultLevel unless an authoritative default becomes available.
Problem
pi/OMP hide the
xhighandmaxthinking tiers for all Command Code models, and silently clamp them tohigh. Root cause: every model is registered withreasoning: truebut nothinkingLevelMap, and pi-ai'sgetSupportedThinkingLevelsgatesxhigh/maxbehind an explicit non-null map entry.Fix
Two changes:
Per-model
thinkingLevelMap— each model registers a map built from its supported reasoning efforts, so pi exposes exactly the tiers the model supports (e.g. DeepSeek V4 Flash showsoff/high/max; Claude Sonnet 5 showslow→max). Unsupported tiers map tonulland are hidden from the UI.reasoning_effortpassthrough — the selected thinking level is forwarded to/alpha/generateasparams.reasoning_effort, matching the official CLI. Theofflevel and unsupported tiers (mapped tonull) are omitted.Where MODEL_EFFORTS comes from
The Provider API (
/provider/v1/models) exposes no effort metadata — every model entry is just{id, name, context_length}. TheMODEL_EFFORTStable insrc/models.ts(27 models) was parsed from the officialcommand-codeCLI 1.7.0 bundle (dist/cli.mjs), which carries areasoningEffortsarray on each model definition.This table will drift as upstream adds models or changes supported efforts. Re-extract by scanning the latest CLI bundle for
id:"..." ... reasoningEfforts:[...]pairs. Models absent from the table get nothinkingLevelMap, preserving pi's defaultoff→hightiers.Evidence that
/alpha/generateacceptsreasoning_effortConfirmed by intercepting the official CLI's real outbound request (NODE_OPTIONS fetch hook wrapping
globalThis.fetch):And the CLI rejects unsupported efforts per-model:
Additional change
COMMANDCODE_MODELS_URLis now resolved inside the entry function instead of at module load, so the env override is honored at call time (was fixed during import, breaking testability).Testing
test-thinking-levels.ts: per-model tier visibility,reasoning_effortpassthrough,offomission,null(unsupported) omission, unknown-model fallback.tsc --noEmitintroduces 0 new errors (3 pre-existing from optionalpi-coding-agentpeerDep).deepseek/deepseek-v4-flashcorrectly yields["off","high","max"]withmaxvisible.