-
Notifications
You must be signed in to change notification settings - Fork 223
Add GLM-5.1 NVFP4 GB200 disaggregated dynamo-sglang MTP config / 添加 GLM-5.1 NVFP4 GB200 分离式 dynamo-sglang MTP 配置 #2115
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
+535
−1
Merged
Changes from all commits
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
36754c1
Add glm5-fp4-gb200-dynamo-sglang-mtp config
RohitNagraj 6fac6b1
Update perf-changelog pr-link for #2115
RohitNagraj 39b6b4b
Pass full CONFIG_FILE selector to srtctl in launch_gb200-nv.sh
RohitNagraj 19de179
Point gb200 MTP config at GLM-5.1 NVFP4 weights
RohitNagraj 037d31b
Use glm5.1 model-prefix for gb200 MTP config (match PR #1780)
RohitNagraj 6da26cd
Align gb200 recipe model.container with image string; drop launcher a…
RohitNagraj a7ee2cb
Merge remote-tracking branch 'origin/main' into glm5-fp4-gb200-dynamo…
RohitNagraj c2d9683
Merge remote-tracking branch 'origin/main' into HEAD
adibarra File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
235 changes: 235 additions & 0 deletions
235
benchmarks/multi_node/srt-slurm-recipes/sglang/glm5/gb200-fp4/glm5-mtp.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,235 @@ | ||
| base: | ||
| name: gb200-fp4-glm5-mtp | ||
| model: | ||
| path: glm-5-fp4 | ||
| container: "lmsysorg/sglang:v0.5.13.post1-cu130" | ||
| precision: fp4 | ||
| identity: | ||
| model: | ||
| repo: nvidia/GLM-5-NVFP4 | ||
| revision: dc54ff55a7e9e71b85db953d8bc22eca894b44c6 | ||
| frameworks: | ||
| dynamo: 1.2.1 | ||
| sglang: 0.5.13.post1 | ||
| resources: | ||
| gpu_type: gb200 | ||
| gpus_per_node: 4 | ||
| frontend: | ||
| type: dynamo | ||
| dynamo: | ||
| version: 1.2.1 | ||
| backend: | ||
| type: sglang | ||
| prefill_environment: | ||
| TORCH_DISTRIBUTED_DEFAULT_TIMEOUT: '1800' | ||
| PYTHONUNBUFFERED: '1' | ||
| DYN_SKIP_SGLANG_LOG_FORMATTING: '1' | ||
| SGLANG_DISAGGREGATION_HEARTBEAT_MAX_FAILURE: '100000' | ||
| SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT: '100000' | ||
| SGLANG_DISAGGREGATION_WAITING_TIMEOUT: '100000' | ||
| MC_TE_METRIC: 'true' | ||
| MC_FORCE_MNNVL: '1' | ||
| NCCL_MNNVL_ENABLE: '1' | ||
| NCCL_CUMEM_ENABLE: '1' | ||
| SGLANG_MOONCAKE_CUSTOM_MEM_POOL: 'True' | ||
| SGLANG_USE_MESSAGE_QUEUE_BROADCASTER: '0' | ||
| SGLANG_DISABLE_TP_MEMORY_INBALANCE_CHECK: '1' | ||
| decode_environment: | ||
| TORCH_DISTRIBUTED_DEFAULT_TIMEOUT: '1800' | ||
| PYTHONUNBUFFERED: '1' | ||
| DYN_SKIP_SGLANG_LOG_FORMATTING: '1' | ||
| SGLANG_DISAGGREGATION_HEARTBEAT_MAX_FAILURE: '100000' | ||
| SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT: '100000' | ||
| SGLANG_DISAGGREGATION_WAITING_TIMEOUT: '100000' | ||
| MC_TE_METRIC: 'true' | ||
| MC_FORCE_MNNVL: '1' | ||
| NCCL_MNNVL_ENABLE: '1' | ||
| NCCL_CUMEM_ENABLE: '1' | ||
| SGLANG_MOONCAKE_CUSTOM_MEM_POOL: 'True' | ||
| SGLANG_USE_MESSAGE_QUEUE_BROADCASTER: '0' | ||
| SGLANG_DISABLE_TP_MEMORY_INBALANCE_CHECK: '1' | ||
| SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK: '512' | ||
| SGLANG_MOE_NVFP4_DISPATCH: '1' | ||
| sglang_config: | ||
| prefill: | ||
| served-model-name: GLM-5-FP4 | ||
| trust-remote-code: true | ||
| quantization: modelopt_fp4 | ||
| kv-cache-dtype: fp8_e4m3 | ||
| disaggregation-mode: prefill | ||
| disaggregation-transfer-backend: nixl | ||
| max-running-requests: 136 | ||
| cuda-graph-max-bs: 136 | ||
| mem-fraction-static: 0.812 | ||
| context-length: 9280 | ||
| chunked-prefill-size: 65536 | ||
| max-prefill-tokens: 16384 | ||
| tensor-parallel-size: 4 | ||
| data-parallel-size: 4 | ||
| expert-parallel-size: 4 | ||
| enable-dp-attention: true | ||
| enable-dp-lm-head: true | ||
| load-balance-method: total_tokens | ||
| nsa-decode-backend: trtllm | ||
| nsa-prefill-backend: trtllm | ||
| moe-runner-backend: flashinfer_trtllm | ||
| fp4-gemm-backend: flashinfer_cutlass | ||
| enable-flashinfer-allreduce-fusion: true | ||
| disable-radix-cache: true | ||
| weight-loader-prefetch-checkpoints: true | ||
| model-loader-extra-config: '{"enable_multithread_load": true}' | ||
| decode: | ||
| served-model-name: GLM-5-FP4 | ||
| trust-remote-code: true | ||
| quantization: modelopt_fp4 | ||
| kv-cache-dtype: fp8_e4m3 | ||
| disaggregation-mode: decode | ||
| disaggregation-transfer-backend: nixl | ||
| speculative-algorithm: EAGLE | ||
| speculative-num-steps: 2 | ||
| speculative-eagle-topk: 1 | ||
| speculative-num-draft-tokens: 3 | ||
| tensor-parallel-size: 4 | ||
| data-parallel-size: 1 | ||
| expert-parallel-size: 1 | ||
| enable-dp-attention: false | ||
| enable-dp-lm-head: false | ||
| load-balance-method: round_robin | ||
| max-running-requests: 22 | ||
| cuda-graph-max-bs: 22 | ||
| chunked-prefill-size: 64 | ||
| context-length: 9280 | ||
| nsa-prefill-backend: trtllm | ||
| nsa-decode-backend: trtllm | ||
| moe-runner-backend: flashinfer_trtllm | ||
| fp4-gemm-backend: flashinfer_cutlass | ||
| mem-fraction-static: 0.918 | ||
| skip-tokenizer-init: true | ||
| stream-interval: 30 | ||
| disable-radix-cache: true | ||
| enable-flashinfer-allreduce-fusion: true | ||
| weight-loader-prefetch-checkpoints: true | ||
| model-loader-extra-config: '{"enable_multithread_load": true}' | ||
| health_check: | ||
| max_attempts: 360 | ||
| interval_seconds: 10 | ||
| benchmark: | ||
| type: sa-bench | ||
| req_rate: inf | ||
| # ################# 8k1k ################# | ||
| zip_override_mtp_8k1k_hightpt: | ||
| name: [8k1k-1p1d_dep16, 8k1k-2p1d_dep16, 8k1k-4p1d_dep16, 8k1k-8p1d_dep16] | ||
| backend: | ||
| decode_environment: | ||
| SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE: '1' | ||
| sglang_config: | ||
| decode: | ||
| cuda-graph-max-bs: [91, 196, 451, 1354] | ||
| data-parallel-size: 16 | ||
| deepep-config: /configs/deepep_config.json | ||
| deepep-mode: low_latency | ||
| enable-dp-attention: true | ||
| enable-dp-lm-head: true | ||
| enable-flashinfer-allreduce-fusion: false | ||
| ep-dispatch-algorithm: static | ||
| ep-num-redundant-experts: 0 | ||
| expert-parallel-size: 16 | ||
| load-balance-method: null | ||
| max-running-requests: [91, 196, 451, 1354] | ||
| mem-fraction-static: 0.858 | ||
| moe-a2a-backend: deepep | ||
| moe-dense-tp-size: 1 | ||
| moe-runner-backend: flashinfer_cutedsl | ||
| speculative-moe-a2a-backend: deepep | ||
| speculative-moe-runner-backend: deep_gemm | ||
| tensor-parallel-size: 16 | ||
| benchmark: | ||
| concurrencies: [[109], [398], [853], [2132]] | ||
| isl: 8192 | ||
| osl: 1024 | ||
| resources: | ||
| decode_nodes: 4 | ||
| decode_workers: 1 | ||
| prefill_nodes: [1, 2, 4, 8] | ||
| prefill_workers: [1, 2, 4, 8] | ||
| zip_override_mtp_8k1k_lowlat: | ||
| name: [8k1k-1p4d_tp4, 8k1k-1p4d_tp4-conc16, 8k1k-1p8d_tp4, 8k1k-1p8d_tp4-conc4, 8k1k-1p16d_tp4, 8k1k-1p16d_tp4-conc1] | ||
| backend: | ||
| sglang_config: | ||
| decode: | ||
| cuda-graph-max-bs: [22, 15, 9, 4, 4, 1] | ||
| max-running-requests: [22, 15, 9, 4, 4, 1] | ||
| benchmark: | ||
| concurrencies: [[104], [82], [87], [47], [74], [26]] | ||
| isl: 8192 | ||
| osl: 1024 | ||
| resources: | ||
| decode_nodes: [4, 4, 8, 8, 16, 16] | ||
| decode_workers: [4, 4, 8, 8, 16, 16] | ||
| prefill_nodes: 1 | ||
| prefill_workers: 1 | ||
| # ################# 1k1k ################# | ||
| zip_override_mtp_1k1k_hightpt: | ||
| name: [1k1k-1p1d_dep16, 1k1k-1p1d_dep16-conc1024, 1k1k-1p1d_dep16-conc256, 1k1k-1p1d_dep32] | ||
| backend: | ||
| decode_environment: | ||
| SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE: '1' | ||
| sglang_config: | ||
| decode: | ||
| context-length: 2112 | ||
| cuda-graph-max-bs: [1627, 958, 239, 1122] | ||
| data-parallel-size: [16, 16, 16, 32] | ||
| deepep-config: /configs/deepep_config.json | ||
| deepep-mode: low_latency | ||
| enable-dp-attention: true | ||
| enable-dp-lm-head: true | ||
| enable-flashinfer-allreduce-fusion: false | ||
| ep-dispatch-algorithm: static | ||
| ep-num-redundant-experts: 0 | ||
| expert-parallel-size: [16, 16, 16, 32] | ||
| max-running-requests: [1627, 958, 239, 1122] | ||
| mem-fraction-static: [0.858, 0.858, 0.858, 0.845] | ||
| moe-a2a-backend: deepep | ||
| moe-dense-tp-size: 1 | ||
| moe-runner-backend: flashinfer_cutedsl | ||
| speculative-moe-a2a-backend: deepep | ||
| speculative-moe-runner-backend: deep_gemm | ||
| tensor-parallel-size: [16, 16, 16, 32] | ||
| prefill: | ||
| context-length: 2112 | ||
| cuda-graph-max-bs: 512 | ||
| disable-cuda-graph: true | ||
| max-running-requests: 512 | ||
| mem-fraction-static: 0.83 | ||
| benchmark: | ||
| concurrencies: [[1742], [1150], [302], [1263]] | ||
| isl: 1024 | ||
| osl: 1024 | ||
| resources: | ||
| decode_nodes: [4, 4, 4, 8] | ||
| decode_workers: 1 | ||
| prefill_nodes: 1 | ||
| prefill_workers: 1 | ||
| zip_override_mtp_1k1k_lowlat: | ||
| name: [1k1k-1p16d_tp4-conc1, 1k1k-1p16d_tp4-conc16, 1k1k-1p16d_tp4-conc4] | ||
| backend: | ||
| sglang_config: | ||
| decode: | ||
| context-length: 2112 | ||
| cuda-graph-max-bs: [1, 16, 4] | ||
| max-running-requests: [1, 16, 4] | ||
| prefill: | ||
| context-length: 2112 | ||
| cuda-graph-max-bs: 512 | ||
| disable-cuda-graph: true | ||
| max-running-requests: 512 | ||
| mem-fraction-static: 0.83 | ||
| benchmark: | ||
| concurrencies: [[26], [343], [94]] | ||
| isl: 1024 | ||
| osl: 1024 | ||
| resources: | ||
| decode_nodes: 16 | ||
| decode_workers: 16 | ||
| prefill_nodes: 1 | ||
| prefill_workers: 1 | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🟡 Both wide-EP decode overrides (
zip_override_mtp_8k1k_hightptat line 129 andzip_override_mtp_1k1k_hightptat line 182) setdeepep-config: /configs/deepep_config.json— a file that isn't vendored in the repo and isn't bind-mounted into the sglang container byrunners/launch_gb200-nv.sh(the onlydefault_mountsblock is populated whenIS_AGENTIC=1and maps/aiperf_mmap_cache//hf_hub_cache, never/configs). Every other sglang recipe in the tree passes an inline JSON string for--deepep-config(e.g.'{"normal_dispatch":{"num_sms":96},"normal_combine":{"num_sms":96}}'), so this new recipe is the sole outlier — either inline the JSON to match the established pattern, or vendor the file undersglang/glm5/gb200-fp4/and mount it explicitly.Extended reasoning...
What the bug is. The two high-throughput decode overrides in
benchmarks/multi_node/srt-slurm-recipes/sglang/glm5/gb200-fp4/glm5-mtp.yamlpassdeepep-config: /configs/deepep_config.jsonto sglang. sglang's--deepep-configargument is treated as inline JSON when the value starts with{, otherwise as a filesystem path to be opened at decode-worker startup. So this recipe requires a real file to exist at/configs/deepep_config.jsoninside the container at startup.Why the file isn't reachable. Verified independently:
find /home/claude/InferenceX -name 'deepep_config*'returns nothing — the file is not vendored anywhere in the repo.runners/launch_gb200-nv.shnever bind-mounts host/configsinto the sglang container on this codepath.DEFAULT_MOUNTS_BLOCKis only populated whenIS_AGENTIC=1(which is not the case for this recipe), and even then it maps/aiperf_mmap_cacheand/hf_hub_cache— never/configs.CONFIG_DIR=/mnt/lustre01/artifacts/sglang-configs/1k1kis exported at the top of the launcher but never referenced anywhere else inrunners/orbenchmarks/— it's dead. It also doesn't point at/configs.sglang/glm5/gb200-fp4/underrecipes/sglang/glm5/gb200-fp4in the srt-slurm workspace, not to/configsinside any container.Pattern break. Grepping the tree, all ~30 existing uses of
--deepep-config(all DSV4 recipes undersglang/deepseek-v4/, and the single-nodedsv4_fp4_b300_sglang_mtp.sh) pass the JSON inline, e.g.deepep-config: '{"normal_dispatch":{"num_sms":88,...},"normal_combine":{"num_sms":88,...}}'. This GLM-5 recipe is the sole outlier using a file path. Siblingglm5/gb300-fp4/recipes don't use--deepep-configat all. The Qwen3.5 recipes that touch/configsuse it for write caches (SGLANG_DG_CACHE_DIR=/configs/deepgemm-cache,FLASHINFER_WORKSPACE_BASE=/configs/flashinfer-cache) — those directories are created by sglang at startup, which is not the same as a config file that must pre-exist.Step-by-step proof of the failure path (assuming the file isn't baked into the container).
nvidia-master.yaml, e.g.8k1k-1p1d_dep16→CONFIG_FILE=recipes/sglang/glm5/gb200-fp4/glm5-mtp.yaml:zip_override_mtp_8k1k_hightpt[0].zip_override_mtp_8k1k_hightptblock ontobase.backend.sglang_config.decode, so the decode command line includes--deepep-config /configs/deepep_config.json.lmsysorg/sglang:v0.5.13.post1-cu130container; the launcher never adds/configstodefault_mounts.--deepep-config. First char is/, not{, so it takes the file-path branch and tries toopen('/configs/deepep_config.json').FileNotFoundErrorand the decode role never comes up — killing every wide-EP sub-config (all four8k1k_hightptrows and all four1k1k_hightptrows, 8 of 17 sub-configs in the sweep, and the primary reason the recipe exists).Why I'm marking this nit. There's a plausible universe where
/configs/deepep_config.jsonis baked intolmsysorg/sglang:v0.5.13.post1-cu130— I can't inspect the container from the repo. If it is, the sweep works; if it isn't, the full-sweep-fail-fast label will surface the concrete failure before merge. Either way the fix is a one-line YAML change and doesn't need to block. But this is still worth cleaning up before merge because (a) it silently ties the recipe to a specific container build, so any image bump could break it invisibly, and (b) it's inconsistent with the established inline-JSON convention used by every DSV4 recipe in the tree.How to fix. Replace the two
deepep-config: /configs/deepep_config.jsonlines at 129 and 182 with an inline JSON string matching what DSV4 uses on GB200 (e.g.deepep-config: '{"normal_dispatch":{"num_sms":96},"normal_combine":{"num_sms":96}}', tuned as appropriate for GLM-5), or vendor the file underbenchmarks/multi_node/srt-slurm-recipes/sglang/glm5/gb200-fp4/and add a bind-mount for it inlaunch_gb200-nv.shon the GLM-5 branch.