Add GLM-5.1 NVFP4 GB200 disaggregated dynamo-sglang MTP config / 添加 GLM-5.1 NVFP4 GB200 分离式 dynamo-sglang MTP 配置#2115
Add GLM-5.1 NVFP4 GB200 disaggregated dynamo-sglang MTP config / 添加 GLM-5.1 NVFP4 GB200 分离式 dynamo-sglang MTP 配置#2115RohitNagraj wants to merge 7 commits into
Conversation
RohitNagraj
requested review from
Ankur-singh,
jgangani and
kedarpotdar-nv
as code owners
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| decode: | ||
| cuda-graph-max-bs: [91, 196, 451, 1354] | ||
| data-parallel-size: 16 | ||
| deepep-config: /configs/deepep_config.json |
There was a problem hiding this comment.
🟡 Both wide-EP decode overrides (zip_override_mtp_8k1k_hightpt at line 129 and zip_override_mtp_1k1k_hightpt at line 182) set deepep-config: /configs/deepep_config.json — a file that isn't vendored in the repo and isn't bind-mounted into the sglang container by runners/launch_gb200-nv.sh (the only default_mounts block is populated when IS_AGENTIC=1 and maps /aiperf_mmap_cache//hf_hub_cache, never /configs). Every other sglang recipe in the tree passes an inline JSON string for --deepep-config (e.g. '{"normal_dispatch":{"num_sms":96},"normal_combine":{"num_sms":96}}'), so this new recipe is the sole outlier — either inline the JSON to match the established pattern, or vendor the file under sglang/glm5/gb200-fp4/ and mount it explicitly.
Extended reasoning...
What the bug is. The two high-throughput decode overrides in benchmarks/multi_node/srt-slurm-recipes/sglang/glm5/gb200-fp4/glm5-mtp.yaml pass deepep-config: /configs/deepep_config.json to sglang. sglang's --deepep-config argument is treated as inline JSON when the value starts with {, otherwise as a filesystem path to be opened at decode-worker startup. So this recipe requires a real file to exist at /configs/deepep_config.json inside the container at startup.
Why the file isn't reachable. Verified independently:
find /home/claude/InferenceX -name 'deepep_config*'returns nothing — the file is not vendored anywhere in the repo.runners/launch_gb200-nv.shnever bind-mounts host/configsinto the sglang container on this codepath.DEFAULT_MOUNTS_BLOCKis only populated whenIS_AGENTIC=1(which is not the case for this recipe), and even then it maps/aiperf_mmap_cacheand/hf_hub_cache— never/configs.CONFIG_DIR=/mnt/lustre01/artifacts/sglang-configs/1k1kis exported at the top of the launcher but never referenced anywhere else inrunners/orbenchmarks/— it's dead. It also doesn't point at/configs.- The srtctl overlay for GLM-5 copies
sglang/glm5/gb200-fp4/underrecipes/sglang/glm5/gb200-fp4in the srt-slurm workspace, not to/configsinside any container.
Pattern break. Grepping the tree, all ~30 existing uses of --deepep-config (all DSV4 recipes under sglang/deepseek-v4/, and the single-node dsv4_fp4_b300_sglang_mtp.sh) pass the JSON inline, e.g. deepep-config: '{"normal_dispatch":{"num_sms":88,...},"normal_combine":{"num_sms":88,...}}'. This GLM-5 recipe is the sole outlier using a file path. Sibling glm5/gb300-fp4/ recipes don't use --deepep-config at all. The Qwen3.5 recipes that touch /configs use it for write caches (SGLANG_DG_CACHE_DIR=/configs/deepgemm-cache, FLASHINFER_WORKSPACE_BASE=/configs/flashinfer-cache) — those directories are created by sglang at startup, which is not the same as a config file that must pre-exist.
Step-by-step proof of the failure path (assuming the file isn't baked into the container).
- Sweep runner picks a wide-EP row from
nvidia-master.yaml, e.g.8k1k-1p1d_dep16→CONFIG_FILE=recipes/sglang/glm5/gb200-fp4/glm5-mtp.yaml:zip_override_mtp_8k1k_hightpt[0]. - srtctl merges the
zip_override_mtp_8k1k_hightptblock ontobase.backend.sglang_config.decode, so the decode command line includes--deepep-config /configs/deepep_config.json. - srtctl schedules the decode workers into the
lmsysorg/sglang:v0.5.13.post1-cu130container; the launcher never adds/configstodefault_mounts. - sglang parses
--deepep-config. First char is/, not{, so it takes the file-path branch and tries toopen('/configs/deepep_config.json'). - If the file isn't baked into the image, this raises
FileNotFoundErrorand the decode role never comes up — killing every wide-EP sub-config (all four8k1k_hightptrows and all four1k1k_hightptrows, 8 of 17 sub-configs in the sweep, and the primary reason the recipe exists).
Why I'm marking this nit. There's a plausible universe where /configs/deepep_config.json is baked into lmsysorg/sglang:v0.5.13.post1-cu130 — I can't inspect the container from the repo. If it is, the sweep works; if it isn't, the full-sweep-fail-fast label will surface the concrete failure before merge. Either way the fix is a one-line YAML change and doesn't need to block. But this is still worth cleaning up before merge because (a) it silently ties the recipe to a specific container build, so any image bump could break it invisibly, and (b) it's inconsistent with the established inline-JSON convention used by every DSV4 recipe in the tree.
How to fix. Replace the two deepep-config: /configs/deepep_config.json lines at 129 and 182 with an inline JSON string matching what DSV4 uses on GB200 (e.g. deepep-config: '{"normal_dispatch":{"num_sms":96},"normal_combine":{"num_sms":96}}', tuned as appropriate for GLM-5), or vendor the file under benchmarks/multi_node/srt-slurm-recipes/sglang/glm5/gb200-fp4/ and add a bind-mount for it in launch_gb200-nv.sh on the GLM-5 branch.
| name: gb200-fp4-glm5-mtp | ||
| model: | ||
| path: glm-5-fp4 | ||
| container: v0.5.13.post1 |
There was a problem hiding this comment.
🟡 Nit: model.container: v0.5.13.post1 in glm5-mtp.yaml:5 is a partial tag while nvidia-master.yaml uses the full image: lmsysorg/sglang:v0.5.13.post1-cu130. AGENTS.md line 148 requires model.container to equal image: (launcher uses it as the container-alias key), and every other sglang recipe in the repo follows that. The PR papers over the mismatch by adding a hardcoded v0.5.13.post1: ${SQUASH_FILE} alias to launch_gb200-nv.sh:424, which works but creates a triple maintenance point on future image bumps. Fix: set model.container: "lmsysorg/sglang:v0.5.13.post1-cu130" in the recipe and drop the v0.5.13.post1: ${SQUASH_FILE} line — the existing "${IMAGE}": ${SQUASH_FILE} entry already covers it.
Extended reasoning...
The convention
AGENTS.md line 148 documents the rule explicitly:
For image bumps,
model.containermust equalimage:, since the launcher uses the latter as the container-alias key.
The launcher-side mechanism this refers to lives in the containers: block written to srtslurm.yaml in launch_gb200-nv.sh:
containers:
dynamo-trtllm: ${SQUASH_FILE}
dynamo-sglang: ${SQUASH_FILE}
"${IMAGE}": ${SQUASH_FILE} # <-- keyed on the full image string from nvidia-master.yamlsrtctl looks up the recipe's model.container value against those keys to find the container squash file. When the recipe uses the same string as nvidia-master.yaml's image:, the existing "${IMAGE}": ${SQUASH_FILE} line covers it automatically — no launcher edit needed.
The mismatch
This PR sets the recipe to a partial tag:
# glm5-mtp.yaml:5
container: v0.5.13.post1while nvidia-master.yaml uses the full image string:
glm5-fp4-gb200-dynamo-sglang-mtp:
image: lmsysorg/sglang:v0.5.13.post1-cu130Because those strings don't match, srtctl's v0.5.13.post1 lookup misses the "${IMAGE}": ${SQUASH_FILE} entry, so the PR added a hardcoded entry alongside it in launch_gb200-nv.sh:424:
containers:
dynamo-trtllm: ${SQUASH_FILE}
dynamo-sglang: ${SQUASH_FILE}
v0.5.13.post1: ${SQUASH_FILE} # <-- new hardcoded per-recipe alias
"${IMAGE}": ${SQUASH_FILE}Grep confirms this is a new pattern: every other sglang recipe under benchmarks/multi_node/srt-slurm-recipes/sglang/ uses the full image string (e.g., lmsysorg/sglang:nightly-dev-cu13-20260528-0abe6a85, lmsysorg/sglang:v0.5.12.post1, lmsysorg/sglang:v0.5.11-cu130), directly matching each recipe's image: in nvidia-master.yaml. The only nearby precedent — dynamo-sglang used by the qwen3.5 recipes — is a framework-level shared alias that predates the launcher's containers block; v0.5.13.post1 is a fresh, single-recipe hardcoded alias.
Step-by-step proof of the maintenance cost
Consider a future image bump from v0.5.13.post1-cu130 → v0.5.14-cu130:
- Bump
image:inconfigs/nvidia-master.yaml(the sweep-generator source of truth). - Bump
container:inbenchmarks/multi_node/srt-slurm-recipes/sglang/glm5/gb200-fp4/glm5-mtp.yaml(srtctl's read). - Extra step introduced by this PR: update the hardcoded
v0.5.13.post1: ${SQUASH_FILE}line inrunners/launch_gb200-nv.sh:424tov0.5.14: ${SQUASH_FILE}.
Miss step 3 and the run silently fails to find the container (or picks up a stale one). Every other sglang recipe only needs steps 1+2 — the "${IMAGE}": ${SQUASH_FILE} entry auto-covers the launcher side because the strings match.
Impact and severity
The sweep functionally works today because the hardcoded launcher alias is in place, so nothing breaks at merge. The cost is future — a hidden coupling that violates the documented convention and adds an easy-to-miss maintenance step. Marking as nit rather than blocking: no runtime failure, just architectural drift.
Fix
- In
benchmarks/multi_node/srt-slurm-recipes/sglang/glm5/gb200-fp4/glm5-mtp.yaml:5:container: "lmsysorg/sglang:v0.5.13.post1-cu130"
- In
runners/launch_gb200-nv.sh:424, drop the added line:The existing- v0.5.13.post1: ${SQUASH_FILE}"${IMAGE}": ${SQUASH_FILE}entry covers it.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28898234005 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28908054083 |
RohitNagraj
changed the title
…-sglang-mtp # Conflicts: # perf-changelog.yaml # runners/launch_gb200-nv.sh
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28911919529 |
1 participant
Adds
glm5-fp4-gb200-dynamo-sglang-mtp, a GB200 NVFP4 disaggregateddynamo-sglang MTP config covering ISL/OSL 8k1k and 1k1k across
high-throughput (wide-EP) and low-latency (per-node TP=4) decode topologies.
benchmarks/multi_node/srt-slurm-recipes/sglang/glm5/gb200-fp4/.glm5.1model-prefix branch tolaunch_gb200-nv.sh.lmsysorg/sglang:v0.5.13.post1-cu130.中文说明
新增
glm5-fp4-gb200-dynamo-sglang-mtp，一个 GB200 NVFP4 分离式 dynamo-sglang MTP 配置，覆盖 ISL/OSL 8k1k 与 1k1k 的高吞吐（wide-EP）和低延迟（每节点 TP=4）解码拓扑。benchmarks/multi_node/srt-slurm-recipes/sglang/glm5/gb200-fp4/下纳入（vendor）GLM-5.1 GB200 sglang recipe。launch_gb200-nv.sh中新增glm5.1model-prefix 分支。lmsysorg/sglang:v0.5.13.post1-cu130。