Skip to content

fix: expose per-model thinking tiers and forward reasoning_effort - #28

Open
m1yuan wants to merge 1 commit into
patlux:mainfrom
m1yuan:fix/expose-thinking-levels
Open

fix: expose per-model thinking tiers and forward reasoning_effort#28
m1yuan wants to merge 1 commit into
patlux:mainfrom
m1yuan:fix/expose-thinking-levels

Conversation

@m1yuan

@m1yuan m1yuan commented Aug 2, 2026

Copy link
Copy Markdown

Problem

pi/OMP hide the xhigh and max thinking tiers for all Command Code models, and silently clamp them to high. Root cause: every model is registered with reasoning: true but no thinkingLevelMap, and pi-ai's getSupportedThinkingLevels gates xhigh/max behind an explicit non-null map entry.

Fix

Two changes:

  1. Per-model thinkingLevelMap — each model registers a map built from its supported reasoning efforts, so pi exposes exactly the tiers the model supports (e.g. DeepSeek V4 Flash shows off/high/max; Claude Sonnet 5 shows lowmax). Unsupported tiers map to null and are hidden from the UI.

  2. reasoning_effort passthrough — the selected thinking level is forwarded to /alpha/generate as params.reasoning_effort, matching the official CLI. The off level and unsupported tiers (mapped to null) are omitted.

Where MODEL_EFFORTS comes from

The Provider API (/provider/v1/models) exposes no effort metadata — every model entry is just {id, name, context_length}. The MODEL_EFFORTS table in src/models.ts (27 models) was parsed from the official command-code CLI 1.7.0 bundle (dist/cli.mjs), which carries a reasoningEfforts array on each model definition.

This table will drift as upstream adds models or changes supported efforts. Re-extract by scanning the latest CLI bundle for id:"..." ... reasoningEfforts:[...] pairs. Models absent from the table get no thinkingLevelMap, preserving pi's default offhigh tiers.

Evidence that /alpha/generate accepts reasoning_effort

Confirmed by intercepting the official CLI's real outbound request (NODE_OPTIONS fetch hook wrapping globalThis.fetch):

POST https://api.commandcode.ai/alpha/generate
params: { ..., reasoning_effort: "max" }   // with --effort max

And the CLI rejects unsupported efforts per-model:

$ command-code --model deepseek/deepseek-v4-flash --effort low
Unknown effort "low". Supported: high, max.

Additional change

COMMANDCODE_MODELS_URL is now resolved inside the entry function instead of at module load, so the env override is honored at call time (was fixed during import, breaking testability).

Testing

  • 5 new tests in test-thinking-levels.ts: per-model tier visibility, reasoning_effort passthrough, off omission, null (unsupported) omission, unknown-model fallback.
  • All existing tests pass (66 total, 0 failures).
  • tsc --noEmit introduces 0 new errors (3 pre-existing from optional pi-coding-agent peerDep).
  • Verified against live Provider API: 27/50 models mapped, deepseek/deepseek-v4-flash correctly yields ["off","high","max"] with max visible.

@m1yuan
m1yuan force-pushed the fix/expose-thinking-levels branch 2 times, most recently from f1bd1b5 to 1e0f6a6 Compare August 2, 2026 03:24
Previously all Command Code models registered reasoning:true with no
thinkingLevelMap, so pi/OMP hid the xhigh/max tiers and clamped them
to high. Now each model registers a thinkingLevelMap built from its
supported efforts (extracted from the official command-code CLI 1.7.0
catalog into MODEL_EFFORTS), exposing exactly the tiers each model
supports — e.g. deepseek-v4-flash shows off/high/max, Claude Sonnet 5
shows low..max.

The selected thinking level is forwarded to /alpha/generate as
params.reasoning_effort, matching the official CLI. The off level and
unsupported tiers (mapped to null) are omitted from the request.

Also resolve COMMANDCODE_MODELS_URL inside the entry function so the
override is honored at call time instead of fixed at module load.

Adds tests for per-model tier visibility, reasoning_effort passthrough,
off/null omission, and the unknown-model fallback.
@patlux
patlux force-pushed the fix/expose-thinking-levels branch from 1e0f6a6 to d36aac7 Compare August 2, 2026 09:02
@patlux
patlux self-requested a review as a code owner August 2, 2026 09:02

@patlux patlux left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tackling this — the underlying pi issue is real and forwarding supported reasoning_effort values is the right direction. I found three blockers that should be addressed before merge:

  1. Keep MODEL_EFFORTS exactly aligned with the authoritative command-code@1.7.0 effort catalog.
  2. Do not infer a per-model default that upstream does not publish.
  3. Emit canonical OMP thinking metadata (mode: "effort" plus efforts) separately from pi's legacy thinkingLevelMap.

I verified that command-code@1.7.0 is currently npm's latest release. Its generated model catalog says models without an effort entry decide their own reasoning depth.

Non-blocking follow-ups:

  • Use mkdtemp()/tmpdir() instead of the shared hard-coded /tmp/commandcode-models-thinking-levels-test.json path, and clean it up in finally.
  • Please add @m1yuan to the Unreleased contributors section.
  • A regression fixture/test containing the exact 1.7.0 effort catalog would help prevent unsupported entries from being introduced again.

Comment thread src/models.ts
"moonshotai/Kimi-K2.5": ["high", "max"],
"moonshotai/Kimi-K2.6": ["high", "max"],
"sakana/fugu-ultra": ["high", "xhigh"],
"tencent/hy3-paid": ["low", "medium", "high"],

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The checked-in table does not match the authoritative command-code@1.7.0 catalog. These five entries have no adjustable effort surface in the published CLI: claude-haiku-4-5-20251001, moonshotai/Kimi-K2.5, moonshotai/Kimi-K2.6, meta/muse-spark-1.1, and tencent/hy3-paid. The CLI's generated reference/models.md marks their Efforts column as ("models without an effort column entry decide their own reasoning depth"). Please remove these entries; otherwise the provider exposes and sends effort values that the official CLI does not support. The remaining 22 entries match the 1.7.0 effort map.

Comment thread src/models.ts
export type ThinkingMetadata = {
thinkingLevelMap: Record<string, string | null>
thinking: {
effortMap: Record<string, string | null>

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OMP 17.2.x expects canonical ThinkingConfig: { mode: "effort", efforts: [...], defaultLevel?, effortMap? }. mode is required, while this effortMap also contains legacy pi-only off/null values that do not belong in OMP's string-valued map. Please keep the legacy thinkingLevelMap for pi, but emit OMP metadata separately, e.g. thinking: { mode: "effort", efforts: [...efforts] }. An OMP identity map is unnecessary because omitted mappings already pass the supported effort through unchanged.

Comment thread src/models.ts
thinking: {
effortMap,
efforts: [...efforts],
defaultLevel: efforts[efforts.length - 2] ?? efforts[0],

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please do not infer defaultLevel from array position. OMP actively applies this value on initial model selection and every model switch, so this makes Claude/GPT-5.6 default to xhigh, Gemini/Grok to medium, etc. The Command Code catalog publishes supported efforts but no per-model default, so this changes runtime behavior and potentially cost without an upstream source. Omit defaultLevel unless an authoritative default becomes available.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants