[WIP] Update vllm B300 Minimax M3#2120
Conversation
wzhao18
requested review from
Ankur-singh,
jgangani and
kedarpotdar-nv
as code owners
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
|
||
| - config-keys: | ||
| - minimaxm3-fp4-b300-vllm | ||
| description: | ||
| - "update vllm image" | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2120 |
There was a problem hiding this comment.
🟡 This PR is materially more than a image bump, but the new perf-changelog entry (perf-changelog.yaml:4623-4628) only says "update vllm image". Per AGENTS.md §"Updating Docker images", each material change should be its own bullet — please also list the new VLLM_MINIMAX_M3_BF16_ROUTER_GEMM=1 env var (with the bf16-router-gate weight patch it enables), the pinned FlashInfer nightly-v0.6.14-20260702 + CuTeDSL split-K gemm patch, and the new --attention_config.indexer_kv_dtype fp8 CLI arg. Also, before dropping the [WIP] marker, please make the PR title bilingual per AGENTS.md (<English title> / <中文标题>).
Extended reasoning...
What the finding is. The perf-changelog entry appended in this PR (perf-changelog.yaml:4623-4628) has a single-bullet description update vllm image, but the PR is doing five distinct things beyond an image bump:
- Sets
VLLM_MINIMAX_M3_BF16_ROUTER_GEMM=1in the recipe (benchmarks/single_node/fixed_seq_len/minimaxm3_fp4_b300.sh:92). - Applies
patches/minimax-m3-bf16-router-gemm.patchto the installed vLLM, which adds that env var tovllm/envs.pyand, when set, switchesMiniMaxM3MoE.gate'sGateLinearparams_dtypefromtorch.float32totorch.bfloat16. That's a real numerical change: instead of upcasting bf16 activations to fp32 and running an fp32 gate, the weights are now rounded fp32→bf16 and the gate runs cuBLAS bf16 × bf16 → fp32. - Installs FlashInfer
0.6.14.dev20260702(nightly-v0.6.14-20260702) explicitly on top of the image, replacing the version baked into the container. - Applies a ~3100-line
flashinfer-cutedsl-splitk-gemm.patchto that installed FlashInfer (new CuTeDSL split-K kernel for low-M MXFP8 gemms). - Adds
--attention_config.indexer_kv_dtype fp8to thevllm serveinvocation (recipe line 112).
Why the convention matters here. AGENTS.md §"Updating Docker images" gives an explicit example showing that an image bump paired with a new env var should be TWO bullets, not one:
description:
- 'Update vLLM image from v0.11.2 to v0.13.0'
- 'Add VLLM_MXFP4_USE_MARLIN=1 environment variable'
The changelog description is what surfaces on inferencex.com next to the perf delta this PR produces. If any of the five items above moves the numbers — and the bf16-router weight rounding in particular is a change to numerical semantics of the routing GEMM, not a no-op — a reader seeing only "update vllm image" will attribute the delta entirely to the image swap. Sibling entries in the same file (e.g. the glm5 configs above at ~4600-4602 and dsr1 at ~4619-4621) all follow the multi-bullet convention.
Step-by-step proof for the bf16 router GEMM entry alone. In patches/minimax-m3-bf16-router-gemm.patch, the GateLinear construction in vllm/models/minimax_m3/nvidia/model.py changes from params_dtype=torch.float32 to params_dtype=torch.bfloat16 if envs.VLLM_MINIMAX_M3_BF16_ROUTER_GEMM else torch.float32. The recipe sets that env var to 1, so on every run the router gate weights are the bf16-rounded version of the checkpoint's fp32 weights. That's a behavior change independent of the image bump, and it's the kind of item AGENTS.md's example says to list as its own bullet.
Bilingual PR title. Separately, AGENTS.md line 7 states "PR and GitHub-issue titles & descriptions must be bilingual — include a Simplified Chinese version in addition to English. Title format: <English title> / <中文标题>. … This applies to every PR and every issue." The current title [WIP] Update vllm B300 Minimax M3 is English-only. One verifier reasonably observed that this is PR metadata on a WIP PR and would normally be finalized when the WIP marker is dropped — that's fine; the note is just a reminder to update the title before requesting review, consistent with how recent MiniMax-M3 image bumps (#2107, #2003) landed.
How to fix. Expand the changelog description to something like:
- 'Update MiniMax-M3 FP4 B300 vLLM image (…→ nightly-2afa3f7e9502…)'
- 'Install pinned FlashInfer nightly-v0.6.14-20260702 and apply the CuTeDSL split-K gemm patch'
- 'Apply MiniMax-M3 bf16-router GEMM patch and enable it via VLLM_MINIMAX_M3_BF16_ROUTER_GEMM=1'
- 'Add --attention_config.indexer_kv_dtype fp8 to vllm serve'
and update the PR title to Update vllm B300 Minimax M3 / <中文标题> before dropping [WIP].
Severity. Nit — this is a documentation completeness issue on a WIP PR, not a runtime defect. Flagging it now so the changelog can be expanded before the [WIP] marker is dropped.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28972702517 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28973507570 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28976585567 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28978689802 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28978821621 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28978901633 |
2 participants
No description provided.