Skip to content

Add GLM-5.1 NVFP4 GB300 disaggregated dynamo-sglang MTP config / 添加 GLM-5.1 NVFP4 GB300 分离式 dynamo-sglang MTP 配置 #2114

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
RohitNagraj wants to merge 7 commits into
base: main
Choose a base branch
from
+305 −10
Open

Add GLM-5.1 NVFP4 GB300 disaggregated dynamo-sglang MTP config / 添加 GLM-5.1 NVFP4 GB300 分离式 dynamo-sglang MTP 配置 #2114

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
cfff85f
Add glm5-fp4-gb300-dynamo-sglang-mtp config
RohitNagraj Jul 7, 2026
58225d0
Update perf-changelog pr-link for #2114
RohitNagraj Jul 7, 2026
189b328
Strip zip_override selector before name-override sed in launch_gb300-…
RohitNagraj Jul 8, 2026
680458b
Point gb300 MTP config at GLM-5.1 NVFP4 weights
RohitNagraj Jul 8, 2026
3abca82
Use glm5.1 model-prefix for gb300 MTP config (match PR #1780 convention)
RohitNagraj Jul 8, 2026
de6750f
Merge remote-tracking branch 'origin/main' into glm5-fp4-gb300-dynamo…
RohitNagraj Jul 8, 2026
929cd38
Skip srtctl preflight for gb300 glm5.1 (weights on compute-only /scra…
RohitNagraj Jul 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
268 changes: 268 additions & 0 deletions configs/nvidia-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11095,6 +11095,274 @@ glm5-fp8-gb200-dynamo-sglang:
ep: 1
dp-attn: false

glm5-fp4-gb300-dynamo-sglang-mtp:

@claude claude Bot Jul 7, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 The PR title and description are English-only, but AGENTS.md line 7 requires bilingual PR titles (<English> / <中文>) and a ## 中文说明 section in the body mirroring the summary. Nit — pure documentation/process issue, no functional impact. Consider updating the title to something like Add GLM-5 NVFP4 GB300 disaggregated dynamo-sglang MTP config / 添加 GLM-5 NVFP4 GB300 分离式 dynamo-sglang MTP 配置 and appending a ## 中文说明 block.

Extended reasoning...

What the rule says. AGENTS.md line 7 states: "PR and GitHub-issue titles & descriptions must be bilingual — include a Simplified Chinese version in addition to English. Title format: <English title> / <中文标题>. In the PR/issue body, follow the English content with its Chinese translation (e.g. a ## 中文说明 section mirroring the summary...)."\n\nWhat this PR has. The PR title is Add GLM-5 NVFP4 GB300 disaggregated dynamo-sglang MTP config — English only, no / <中文> suffix. The body is English-only bullets describing the config, image, and changelog entry; there is no ## 中文说明 section (or any Chinese content) mirroring the summary.\n\nHow this manifests. This is purely a process/documentation violation. The recipe itself (glm5-fp4-gb300-dynamo-sglang-mtp), the v0.5.13.post1 container alias in runners/launch_gb300-nv.sh, and the perf-changelog.yaml entry are all correctly wired — there is no runtime, build, CI, or sweep impact. It's a review/discoverability concern only.\n\nEnforcement precedent. Recent recipe-add PRs merged to main demonstrate the convention is real and achievable: #2107 (Minimax-M3-MXFP8 Eagle3 backend override) carries ... /Minimax-M3-MXFP8 Eagle3草稿模型的注意力后端固定为 TRITON_ATTN以提高性能 and #2101 (gpt-oss-120b deprecation) carries ... / 弃用 gpt-oss-120b 基准测试配置. Enforcement is not perfectly uniform (e.g. #2110, #2100 lack the Chinese half), but the rule is on the books and neighboring PRs of the same shape follow it.\n\nStep-by-step proof.\n\n1. Read AGENTS.md line 7 → rule: title format <English> / <中文>, body needs ## 中文说明 section.\n2. Read PR #2114 title from <pr-metadata>: Add GLM-5 NVFP4 GB300 disaggregated dynamo-sglang MTP config → no / separator, no Chinese characters (scanning for CJK returns no matches).\n3. Read PR #2114 body → four bullets in English describing the config, image, and changelog entry. Search body for 中文说明 or any CJK character → none present.\n4. Compare against git log entries for #2107 and #2101 — both have Chinese titles → rule is actively followed by peers.\n5. Conclusion: title and body both violate AGENTS.md §Bilingualism convention.\n\nHow to fix. Two edits, both to PR metadata (no code changes):\n\n- Rename the PR title to append a Chinese translation, e.g.: Add GLM-5 NVFP4 GB300 disaggregated dynamo-sglang MTP config / 添加 GLM-5 NVFP4 GB300 分离式 dynamo-sglang MTP 配置.\n- Append a ## 中文说明 section to the PR body that mirrors the English summary (glm5-fp4-gb300-dynamo-sglang-mtp recipe, ISL/OSL 8k1k+1k1k coverage, high-throughput wide-EP + low-latency TP=4 decode topologies, v0.5.13.post1 container alias in launch_gb300-nv.sh, image lmsysorg/sglang:v0.5.13.post1-cu130, and the perf-changelog entry).\n\nNo files in this diff need to change.

image: lmsysorg/sglang:v0.5.13.post1-cu130
model: nvidia/GLM-5.1-NVFP4
model-prefix: glm5.1
runner: gb300-nv
precision: fp4
framework: dynamo-sglang
multinode: true
disagg: true
scenarios:
fixed-seq-len:
# ---------- 8k1k high-throughput (wide-EP TP=16 decode, MTP) ----------
- isl: 8192
osl: 1024
search-space:
# 4p1d dep16. 6 nodes (2P + 4D).
- spec-decoding: "mtp"
conc-list: [364]
prefill:
num-worker: 4
tp: 2
ep: 2
dp-attn: true
additional-settings:
- "CONFIG_FILE=recipes/gb300-fp4/glm5-mtp.yaml:zip_override_mtp_8k1k_hightpt[0]"
decode:
num-worker: 1
tp: 16
ep: 16
dp-attn: true
# 8p1d dep16. 8 nodes (4P + 4D).
- spec-decoding: "mtp"
conc-list: [512]
prefill:
num-worker: 8
tp: 2
ep: 2
dp-attn: true
additional-settings:
- "CONFIG_FILE=recipes/gb300-fp4/glm5-mtp.yaml:zip_override_mtp_8k1k_hightpt[1]"
decode:
num-worker: 1
tp: 16
ep: 16
dp-attn: true
# ---------- 8k1k low-latency (per-node TP=4 decode workers, MTP) ----------
- isl: 8192
osl: 1024
search-space:
# 1p2d tp4. 3 nodes.
- spec-decoding: "mtp"
conc-list: [66]
prefill:
num-worker: 1
tp: 2
ep: 2
dp-attn: true
additional-settings:
- "CONFIG_FILE=recipes/gb300-fp4/glm5-mtp.yaml:zip_override_mtp_8k1k_lowlat[0]"
decode:
num-worker: 2
tp: 4
ep: 1
dp-attn: false
# 1p2d tp4, conc16. 3 nodes.
- spec-decoding: "mtp"
conc-list: [41]
prefill:
num-worker: 1
tp: 2
ep: 2
dp-attn: true
additional-settings:
- "CONFIG_FILE=recipes/gb300-fp4/glm5-mtp.yaml:zip_override_mtp_8k1k_lowlat[1]"
decode:
num-worker: 2
tp: 4
ep: 1
dp-attn: false
# 1p4d tp4. 5 nodes.
- spec-decoding: "mtp"
conc-list: [55]
prefill:
num-worker: 1
tp: 2
ep: 2
dp-attn: true
additional-settings:
- "CONFIG_FILE=recipes/gb300-fp4/glm5-mtp.yaml:zip_override_mtp_8k1k_lowlat[2]"
decode:
num-worker: 4
tp: 4
ep: 1
dp-attn: false
# 1p4d tp4, conc4. 5 nodes.
- spec-decoding: "mtp"
conc-list: [23]
prefill:
num-worker: 1
tp: 2
ep: 2
dp-attn: true
additional-settings:
- "CONFIG_FILE=recipes/gb300-fp4/glm5-mtp.yaml:zip_override_mtp_8k1k_lowlat[3]"
decode:
num-worker: 4
tp: 4
ep: 1
dp-attn: false
# 1p8d tp4. 9 nodes.
- spec-decoding: "mtp"
conc-list: [44]
prefill:
num-worker: 1
tp: 2
ep: 2
dp-attn: true
additional-settings:
- "CONFIG_FILE=recipes/gb300-fp4/glm5-mtp.yaml:zip_override_mtp_8k1k_lowlat[4]"
decode:
num-worker: 8
tp: 4
ep: 1
dp-attn: false
# 1p8d tp4, conc1. 9 nodes.
- spec-decoding: "mtp"
conc-list: [12]
prefill:
num-worker: 1
tp: 2
ep: 2
dp-attn: true
additional-settings:
- "CONFIG_FILE=recipes/gb300-fp4/glm5-mtp.yaml:zip_override_mtp_8k1k_lowlat[5]"
decode:
num-worker: 8
tp: 4
ep: 1
dp-attn: false
# ---------- 1k1k high-throughput (wide-EP decode, MTP) ----------
- isl: 1024
osl: 1024
search-space:
# 1p1d dep16. 5 nodes.
- spec-decoding: "mtp"
conc-list: [773]
prefill:
num-worker: 1
tp: 2
ep: 2
dp-attn: true
additional-settings:
- "CONFIG_FILE=recipes/gb300-fp4/glm5-mtp.yaml:zip_override_mtp_1k1k_hightpt[0]"
decode:
num-worker: 1
tp: 16
ep: 16
dp-attn: true
# 1p1d dep16, conc512. 5 nodes.
- spec-decoding: "mtp"
conc-list: [578]
prefill:
num-worker: 1
tp: 2
ep: 2
dp-attn: true
additional-settings:
- "CONFIG_FILE=recipes/gb300-fp4/glm5-mtp.yaml:zip_override_mtp_1k1k_hightpt[1]"
decode:
num-worker: 1
tp: 16
ep: 16
dp-attn: true
# 1p1d dep32. 9 nodes.
- spec-decoding: "mtp"
conc-list: [677]
prefill:
num-worker: 1
tp: 2
ep: 2
dp-attn: true
additional-settings:
- "CONFIG_FILE=recipes/gb300-fp4/glm5-mtp.yaml:zip_override_mtp_1k1k_hightpt[2]"
decode:
num-worker: 1
tp: 32
ep: 32
dp-attn: true
# 2p1d dep16. 5 nodes.
- spec-decoding: "mtp"
conc-list: [2048]
prefill:
num-worker: 2
tp: 2
ep: 2
dp-attn: true
additional-settings:
- "CONFIG_FILE=recipes/gb300-fp4/glm5-mtp.yaml:zip_override_mtp_1k1k_hightpt[3]"
decode:
num-worker: 1
tp: 16
ep: 16
dp-attn: true
# 2p1d dep16, conc1024. 5 nodes.
- spec-decoding: "mtp"
conc-list: [1362]
prefill:
num-worker: 2
tp: 2
ep: 2
dp-attn: true
additional-settings:
- "CONFIG_FILE=recipes/gb300-fp4/glm5-mtp.yaml:zip_override_mtp_1k1k_hightpt[4]"
decode:
num-worker: 1
tp: 16
ep: 16
dp-attn: true
# ---------- 1k1k low-latency (per-node TP=4 decode workers, MTP) ----------
- isl: 1024
osl: 1024
search-space:
# 1p8d tp4, conc1. 9 nodes.
- spec-decoding: "mtp"
conc-list: [13]
prefill:
num-worker: 1
tp: 2
ep: 2
dp-attn: true
additional-settings:
- "CONFIG_FILE=recipes/gb300-fp4/glm5-mtp.yaml:zip_override_mtp_1k1k_lowlat[0]"
decode:
num-worker: 8
tp: 4
ep: 1
dp-attn: false
# 1p8d tp4, conc16. 9 nodes.
- spec-decoding: "mtp"
conc-list: [162]
prefill:
num-worker: 1
tp: 2
ep: 2
dp-attn: true
additional-settings:
- "CONFIG_FILE=recipes/gb300-fp4/glm5-mtp.yaml:zip_override_mtp_1k1k_lowlat[1]"
decode:
num-worker: 8
tp: 4
ep: 1
dp-attn: false
# 1p8d tp4, conc4. 9 nodes.
- spec-decoding: "mtp"
conc-list: [44]
prefill:
num-worker: 1
tp: 2
ep: 2
dp-attn: true
additional-settings:
- "CONFIG_FILE=recipes/gb300-fp4/glm5-mtp.yaml:zip_override_mtp_1k1k_lowlat[2]"
decode:
num-worker: 8
tp: 4
ep: 1
dp-attn: false

glm5-fp8-gb300-dynamo-sglang:
image: lmsysorg/sglang:v0.5.11-cu130
model: zai-org/GLM-5-FP8
Expand Down
7 changes: 7 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4589,3 +4589,10 @@
- "Image: lmsysorg/sglang:v0.5.12"
- "14 topologies across 1k/1k and 8k/1k: prefill TP8 STP + decode wide-EP (DEP16/DEP32 high-throughput) and per-node TP8 low-latency, recipes under benchmarks/multi_node/srt-slurm-recipes/sglang/glm5/gb200-fp8/"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/1895

- config-keys:
- glm5-fp4-gb300-dynamo-sglang-mtp
description:
- "Add GLM-5.1 NVFP4 GB300 disaggregated dynamo-sglang MTP config covering 8k1k and 1k1k, high-throughput and low-latency topologies"
- "Image: lmsysorg/sglang:v0.5.13.post1-cu130"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2114
40 changes: 30 additions & 10 deletions runners/launch_gb300-nv.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,11 @@ elif [[ $MODEL_PREFIX == "glm5" && $PRECISION == "fp4" && $FRAMEWORK == "dynamo-
export SERVED_MODEL_NAME="glm-5-nvfp4"
export MODEL_PATH=/scratch/models/GLM-5-NVFP4
export SRT_SLURM_MODEL_PREFIX="nvidia/GLM-5-NVFP4"
elif [[ $MODEL_PREFIX == "glm5.1" && $PRECISION == "fp4" ]]; then
# SRT_SLURM_MODEL_PREFIX matches the model.path alias ("glm-5-fp4")
# in our GLM-5.1 sglang recipes.
export MODEL_PATH=/scratch/models/GLM-5.1-NVFP4
export SRT_SLURM_MODEL_PREFIX="glm-5-fp4"
elif [[ $MODEL_PREFIX == "glm5" && $PRECISION == "fp4" ]]; then
export MODEL_PATH=/scratch/models/GLM-5-NVFP4
export SRT_SLURM_MODEL_PREFIX="glm-5-fp4"
Expand Down Expand Up @@ -164,6 +169,12 @@ elif [[ $FRAMEWORK == "dynamo-sglang" && $MODEL_PREFIX == "glm5" ]]; then
mkdir -p recipes/sglang/glm5/gb300-fp4
cp -rT "$GITHUB_WORKSPACE/benchmarks/multi_node/srt-slurm-recipes/sglang/glm5/gb300-fp4" recipes/sglang/glm5/gb300-fp4
fi
elif [[ $FRAMEWORK == "dynamo-sglang" && $MODEL_PREFIX == "glm5.1" ]]; then
# GLM-5.1 MTP recipe (recipes/gb300-fp4/glm5-mtp.yaml) lives on
# NVIDIA/srt-slurm:main — check it out; no in-repo overlay needed.
git clone https://github.com/NVIDIA/srt-slurm.git "$SRT_REPO_DIR"
cd "$SRT_REPO_DIR"
git checkout main
elif [[ $FRAMEWORK == "dynamo-sglang" && $MODEL_PREFIX == "qwen3.5" ]]; then
# Overlay our version-controlled Qwen3.5 recipes onto the submission branch.
git clone https://github.com/NVIDIA/srt-slurm.git "$SRT_REPO_DIR"
Expand Down Expand Up @@ -255,6 +266,7 @@ containers:
dynamo-trtllm: ${SQUASH_FILE}
dynamo-sglang: ${SQUASH_FILE}
v0.5.11: ${SQUASH_FILE}
v0.5.13.post1: ${SQUASH_FILE}
"${IMAGE}": ${SQUASH_FILE}
nginx-sqsh: ${NGINX_SQUASH_FILE}
use_segment_sbatch_directive: false
Expand All @@ -277,20 +289,28 @@ if [[ -z "$CONFIG_FILE" ]]; then
exit 1
fi

# Override the job name in the config file with the runner name
sed -i "s/^name:.*/name: \"${RUNNER_NAME}\"/" "$CONFIG_FILE"

# --no-preflight is only safe on the agentic path, where the recipe
# resolves model.path to /scratch (compute-node-only NVMe) and the
# srtctl process running on the GHA runner pod can't see it. Fixed-
# seq-len recipes still resolve model.path to an NFS-visible location
# where the precheck is a useful sanity guard, so keep enforcement on
# for them.
# Override the job name in the config file with the runner name.
# CONFIG_FILE may carry a ":zip_override_...[i]" selector suffix that only
# `srtctl apply -f` parses; strip it to the real path for the sed. srtctl
# below still receives the full CONFIG_FILE (with selector).
CONFIG_PATH="${CONFIG_FILE%%:*}"
sed -i "s/^name:.*/name: \"${RUNNER_NAME}\"/" "$CONFIG_PATH"

# --no-preflight skips srtctl's pre-submit model-path stat, which runs on
# the GHA runner host (im-gb300-login-02, an x86 login node). It's required
# whenever model.path resolves to the node-local /scratch NVMe that the login
# node can't see:
# - the agentic path (DSv4-Pro checkpoint), and
# - glm5.1, whose GLM-5.1-NVFP4 weights are prestaged on the compute-node
# /scratch/models.
# The engine still fails loudly at runtime if the path is genuinely missing on
# the compute node. Other fixed-seq-len recipes resolve model.path to a
# login-visible location, so keep the precheck enforced for them.
SRTCTL_APPLY_ARGS=(
-f "$CONFIG_FILE"
--tags "gb300,${MODEL_PREFIX},${PRECISION},${ISL}x${OSL},infmax-$(date +%Y%m%d)"
)
if [[ "$IS_AGENTIC" == "1" ]]; then
if [[ "$IS_AGENTIC" == "1" || "$MODEL_PREFIX" == "glm5.1" ]]; then
SRTCTL_APPLY_ARGS+=(--no-preflight)
fi
if [[ -n "$SRTCTL_SETUP_SCRIPT" ]]; then
Expand Down
Loading