[NV] llm-d-vllm: Add llm-d to the InferenceX benchmarking framework#2050
[NV] llm-d-vllm: Add llm-d to the InferenceX benchmarking framework#2050ezrasilvera wants to merge 12 commits into
Conversation
ezrasilvera
requested review from
Ankur-singh,
jgangani and
kedarpotdar-nv
as code owners
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
2 similar comments
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| # | ||
| # This is currently not used as we build an use an image that contains | ||
| # all binaries |
There was a problem hiding this comment.
🟡 The comment on lines 10-12 says "This is currently not used as we build an use an image that contains all binaries," but benchmarks/multi_node/llm-d/job.slurm (lines 236-241 of the pyxis branch) explicitly sources this file and uses LLMD_BIN_DIR to conditionally bind-mount epp/pd-sidecar/envoy — so the file IS wired into the runtime as an optional override. Also a typo: "build an use" → "build and use". Nit — no runtime impact, but the stale comment will mislead maintainers into thinking they can delete the file.
Extended reasoning...
What the comment claims vs. what the code does
Lines 10-12 of benchmarks/llm-d/binaries.env state:
This is currently not used as we build an use an image that contains all binaries
However, the pyxis branch of benchmarks/multi_node/llm-d/job.slurm explicitly sources this file and consumes LLMD_BIN_DIR to build optional bind-mounts:
[[ -f "${DI_REPO_DIR}/benchmarks/llm-d/binaries.env" ]] && \
source "${DI_REPO_DIR}/benchmarks/llm-d/binaries.env"
for _bin in epp pd-sidecar envoy; do
if [[ -n "${LLMD_BIN_DIR:-}" && -x "${LLMD_BIN_DIR}/${_bin}" ]]; then
PYXIS_MOUNTS+=",${LLMD_BIN_DIR}/${_bin}:/usr/local/bin/${_bin}:ro"
...
fi
doneSo binaries.env is wired into the runtime — it acts as the single source of truth for LLMD_BIN_DIR (defaulted to /mnt/lustre01/users-public/sa-shared/llm-d-bins), which the pyxis path uses to swap in mounted binaries and skip a combined-image rebuild.
Step-by-step proof
- Author enables the pyxis path (the default on GB200 via
LLMD_CONTAINER_ENGINE=pyxisset inrunners/launch_gb200-nv.sh). job.slurmreaches the pyxis branch and sourcesbinaries.env, populatingLLMD_BIN_DIR.- If a user has run
extract-binaries.sh(which also sources this file), the three binaries exist under$LLMD_BIN_DIRand get bind-mounted into every container. - This is the entire point of the mounted-binary path documented in
extract-binaries.shand the Dockerfile — running a stockvllm/vllm-openaiimage without rebuilding the combined image.
The comment therefore contradicts both the sibling files (extract-binaries.sh, Dockerfile, envoy.yaml docstrings) that describe binaries.env as the single source of truth, and job.slurm itself.
Addressing the counter-argument
One could argue the comment is "arguably accurate for the DEFAULT deployment path" — in the current recipe the combined image bakes binaries in, so the -x check makes the mount loop a no-op unless the operator has explicitly run extract-binaries.sh. That's a fair characterization of when the file has effect, but "currently not used" is stronger than "currently a no-op by default" and it flatly contradicts the sourcing on job.slurm:236. A future maintainer trying to trim "unused" scaffolding would delete this file and quietly break the mounted-binary path.
Impact and fix
No runtime impact — the code works today. The concern is purely documentation drift in a new file this PR is introducing. Fix in one edit:
# This file is sourced by benchmarks/multi_node/llm-d/job.slurm to
# optionally bind-mount extracted binaries in place of the combined
# image's baked-in copies (see extract-binaries.sh). It is a no-op
# by default (the -x check fails when the binaries have not been
# extracted); the Dockerfile also reads the same URLs when building
# and using an image that contains all binaries.
Also fix the "build an use" → "build and use" typo while you're in there.
| ### Why one entry per *leader* (not per node) | ||
|
|
||
| In the wide-EP guide each instance is a single vLLM engine that spans | ||
| multiple nodes via `--data-parallel-hybrid-lb`. With hybrid-lb, the | ||
| leader pod (`LWS_WORKER_INDEX=0`) accepts external traffic and | ||
| distributes it internally across the local DP ranks; in our LWS-free | ||
| SLURM mapping, the prefill-leader and decode-leader are the only nodes | ||
| addressable from outside. Adding an entry per worker would cause EPP to | ||
| route directly to a worker, bypassing the engine's internal load | ||
| balancing. | ||
|
|
||
| If we later want to expose all pods of an instance (the alternative | ||
| hybrid-lb interpretation: external LB across nodes too), we can extend | ||
| the loop in `server.sh` to emit one entry per `IPS[i]` in the prefill | ||
| range and one per `IPS[i]` in the decode range, all carrying the same | ||
| role label. EPP then load-balances across them via `random-picker`. |
There was a problem hiding this comment.
🟡 The README section "Why one entry per leader (not per node)" (lines 80-89) plus the example endpoints.yaml at lines 57-69 both document per-leader emission, but server.sh add_role() (lines 334-343) actually emits ONE endpoint per node — its own comment says "ONE endpoint per node at base_port (hybrid LB: ...)". The lines 91-95 "future work" section describing per-node emission as hypothetical has already been implemented. Docs-only staleness (per-node is the intended behavior — see server.sh:289-291 and its consistency with the multi-node coordinator health poll in lines 519-535), but future maintainers may misdiagnose whether current behavior is intentional. Please update the README to reflect actual per-node emission, or add a note explaining why the leader-only text was preserved.
Extended reasoning...
What the bug is
The new file benchmarks/multi_node/llm-d/README.md (added by this PR) describes the endpoints.yaml file-discovery contract as emitting one entry per leader. Concretely:
- Line 55: "…writes
/tmp/endpoints.yamlinside the container with one entry per leader" - Lines 57-69: the example YAML shows only
prefill-0(PREFILL_LEADER_IP) anddecode-0(DECODE_LEADER_IP) - Lines 80-89: an entire section titled "Why one entry per leader (not per node)" justifies leader-only emission and warns: "Adding an entry per worker would cause EPP to route directly to a worker, bypassing the engine's internal load balancing"
- Lines 91-95: describes per-node emission as a hypothetical future extension ("we can extend the loop … to emit one entry per IPS[i]")
But the code at benchmarks/multi_node/llm-d/server.sh:334-348 (the add_role() block that writes endpoints.yaml) does exactly the opposite:
def add_role(role, ips, base_port):
# ONE endpoint per node at base_port (hybrid LB: the node's api-server /
# sidecar internally load-balances its local DP ranks).
for i, ip in enumerate(ips):
endpoints.append({'name': f'{role}-{i}', ...})
add_role('prefill', prefill_ips, VLLM_PORT) # prefill_ips = all_ips[:pn]
add_role('decode', decode_ips, SIDECAR_PORT) # decode_ips = all_ips[pn:pn + dn]The inline comment at server.sh:339-340 literally says the opposite of the README section header: "ONE endpoint per node at base_port".
Step-by-step proof
Take the new high-tpt 2P+1D key (dsv4-fp4-gb200-llm-d-vllm-high-tpt-megamoe, configs/nvidia-master.yaml). Topology: PREFILL_NODES=4, DECODE_NODES=2, PREFILL_WORKERS=2.
submit.shsbatches 6 nodes.job.slurmcollects their IPs in rank order intoALL_IPS = [P0,P1,P2,P3,D0,D1].- On the decode leader,
server.shruns the coordinator Python block:prefill_ips = all_ips[:4] = [P0, P1, P2, P3]decode_ips = all_ips[4:6] = [D0, D1]
add_role('prefill', prefill_ips, 8200)iterates 4 times → emitsprefill-0..prefill-3.add_role('decode', decode_ips, 8000)iterates 2 times → emitsdecode-0..decode-1.- Final endpoints.yaml has 6 entries — not the 2 (prefill-0 + decode-0) the README example shows.
For the 1P+1D mid-curve key it emits 1 prefill + 2 decode entries (still per-node, not per-leader).
Why this is intentional (i.e. the code is right, docs are stale)
Several signals confirm per-node emission is the deliberate design, not a bug:
- server.sh:289-291 comment explicitly justifies it: "Each decode node runs its own sidecar (SIDECAR_PORT -> local decode vLLM), and endpoints.yaml lists one decode endpoint per node so EPP fans out across all decode ranks."
- server.sh:519-535 (the decode coordinator's cross-node prefill health poll) iterates over
_prefill_ips=( "${_ALL_IPS[@]:0:${PREFILL_NODES}}" )— polling every prefill node, not just the leader. That poll is only meaningful if all prefill nodes are individually addressable endpoints. --api-server-countis set on every EP-enabled node (server.sh:236-240), plus--data-parallel-hybrid-lb— meaning each node has its own api-server binding, so per-node endpoints are actually the correct interpretation of hybrid-lb.- PREFILL_WORKERS=2 in the high-tpt key requires per-node emission: with 2 independent DP engines across 4 prefill nodes, per-leader emission would only expose one of the two engines.
Impact
Zero runtime impact — code is self-consistent and produces the intended routing behavior. But a future maintainer reading the README's "Why one entry per leader (not per node)" section will:
- Believe the current behavior is per-leader when it isn't.
- Reference the warning about "EPP would bypass internal load balancing" and (incorrectly) conclude the current implementation is broken.
- Not realize the "future work" paragraph at lines 91-95 has already been implemented.
Suggested fix
Update benchmarks/multi_node/llm-d/README.md:
- Line 55: change "one entry per leader" → "one entry per node".
- Lines 57-69: expand the example YAML to show all prefill and decode nodes (e.g.
prefill-0..prefill-N-1,decode-0..decode-M-1). - Lines 80-89: rewrite the "Why one entry per leader" section to explain why per-node is used (hybrid-lb + independent api-servers + PREFILL_WORKERS multi-engine).
- Lines 91-95: delete or rewrite the "future work" paragraph since per-node emission is what the code already does. The leader-only mode could be described here as the hypothetical alternative if desired.
There was a problem hiding this comment.
Fixed in d7e4180 - README now documents one endpoint per node (not per leader) to match server.sh add_role(), with corrected example ports (prefill 8200 / decode 8000) and a rewritten rationale section.
| - "Add DeepSeek-V4-Pro FP4 GB200 llm-d-vllm disaggregated multinode benchmarks" | ||
| - "Mid-curve 1P+1D DEP8 (conc 256/512/1024)" | ||
| - "High-tpt 2P+1D DEP8 (conc 4096)" | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX |
There was a problem hiding this comment.
🟡 The new perf-changelog entry has pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX (literal XXX placeholder at perf-changelog.yaml:4444). Every other entry in this file resolves to a real PR number — this one alone will 404 if merged as-is. Fix by replacing XXX with 2050 before merge. (The author already listed this under 'Before merge' in the PR description, so filing as a friendly nit rather than a blocker.)
Extended reasoning...
What the bug is. The perf-changelog entry appended by this PR (perf-changelog.yaml:4441-4444) carries a literal pull/XXX placeholder rather than the real PR number:
- config-keys:
- dsv4-fp4-gb200-llm-d-vllm-mid-curve-megamoe
- dsv4-fp4-gb200-llm-d-vllm-high-tpt-megamoe
description:
- "Add DeepSeek-V4-Pro FP4 GB200 llm-d-vllm disaggregated multinode benchmarks"
- "Mid-curve 1P+1D DEP8 (conc 256/512/1024)"
- "High-tpt 2P+1D DEP8 (conc 4096)"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXXWhy the current diff is inconsistent. Every other entry in perf-changelog.yaml resolves to a real PR (the immediately preceding entry ends with pull/2001; earlier entries reference pull/1990, pull/1978, etc.). The pr-link contract of this file is to trace which PR introduced which benchmark set — a literal XXX breaks that contract.
Impact if merged as-is. Clicking the link in the changelog will hit https://github.com/SemiAnalysisAI/InferenceX/pull/XXX, which resolves to no PR (GitHub returns 404 for numeric paths that don't exist, or worse silently redirects). No functionality is affected — the sweep still runs, the config keys still work — but the diagnostic "which PR introduced this row?" trail is broken for the two new dsv4-fp4-gb200-llm-d-vllm keys.
Why existing checks don't catch it. There is no linter validating perf-changelog pr-link values against real PR numbers, so nothing stops the placeholder from shipping. The PR description's 'Before merge' checklist explicitly lists this: 'Canonicalize perf-changelog.yaml pr-link from pull/XXX to this PR's number' — the author has already flagged it as a known merge-blocker to canonicalize.
Step-by-step proof.
- Open
perf-changelog.yamlat line 4444 → observepr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX. grep -c 'pull/XXX' perf-changelog.yaml→ 1 (only this new entry).grep -oE 'pull/[0-9]+' perf-changelog.yaml | tail -3→ all resolve to real numeric PRs (2001, 1990, 1978).- Attempt
https://github.com/SemiAnalysisAI/InferenceX/pull/XXXin a browser → GitHub 404. - Expected value:
pull/2050(this PR's number, per the PR metadata).
Fix. One-character-scope edit: replace XXX with 2050 on perf-changelog.yaml:4444. No other files need to change.
There was a problem hiding this comment.
Fixed in d7e4180 - pr-link canonicalized from pull/XXX to pull/2050.
- llm-d/README.md: document endpoints.yaml as one entry per node (not per leader) to match server.sh add_role(); fix example ports (prefill 8200, decode 8000) and rewrite the rationale section (C2) - perf-changelog.yaml: canonicalize pr-link XXX -> 2050 (C4) Signed-off-by: Ezra Silvera <ezra@il.ibm.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28702992096 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28703886323 |
…ent (C1) Bump the EPP and P/D routing-sidecar source images from the moving -dev:main tags to release :v0.9.0 in both binaries.env and the combined-image Dockerfile, so a rebuild is reproducible. Verified both tags ship linux/arm64 and still expose /app/epp and /app/pd-sidecar (statically-linked aarch64). Envoy pin unchanged. Also reword the binaries.env header (PR #2050 review C1): the old comment claimed the file is 'not used', but job.slurm still sources it and conditionally bind-mounts the binaries via LLMD_BIN_DIR. Clarify it is an optional mounted-binary override, a no-op by default, and fix the 'build an use' -> 'build and use' typo. Signed-off-by: Ezra Silvera <ezra@il.ibm.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28707148515 |
Add dsv4-fp4-gb200-llm-d-vllm-max-tpt-megamoe: 3 prefill (DEP8) + 1 decode (DEP8) at conc=4096, mirroring NVIDIA/srt-slurm disagg-gb200-max-tpt-megamoe (engine-identical to high-tpt; only prefill scale-out 2->3), so it reuses the same CONFIG_FILE. Validated standalone via e2e run 28711505502 at 8797 tok/s/GPU (median TTFT 58 ms) - above high-tpt's 8241 tok/s/GPU, since the extra prefill instance relieves the conc=4096 prefill bottleneck. Adds a matching perf-changelog entry under PR #2050. Signed-off-by: Ezra Silvera <ezra@il.ibm.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28714892207 |
- llm-d/README.md: document endpoints.yaml as one entry per node (not per leader) to match server.sh add_role(); fix example ports (prefill 8200, decode 8000) and rewrite the rationale section (C2) - perf-changelog.yaml: canonicalize pr-link XXX -> 2050 (C4) Signed-off-by: Ezra Silvera <ezra@il.ibm.com>
…ent (C1) Bump the EPP and P/D routing-sidecar source images from the moving -dev:main tags to release :v0.9.0 in both binaries.env and the combined-image Dockerfile, so a rebuild is reproducible. Verified both tags ship linux/arm64 and still expose /app/epp and /app/pd-sidecar (statically-linked aarch64). Envoy pin unchanged. Also reword the binaries.env header (PR #2050 review C1): the old comment claimed the file is 'not used', but job.slurm still sources it and conditionally bind-mounts the binaries via LLMD_BIN_DIR. Clarify it is an optional mounted-binary override, a no-op by default, and fix the 'build an use' -> 'build and use' typo. Signed-off-by: Ezra Silvera <ezra@il.ibm.com>
Add dsv4-fp4-gb200-llm-d-vllm-max-tpt-megamoe: 3 prefill (DEP8) + 1 decode (DEP8) at conc=4096, mirroring NVIDIA/srt-slurm disagg-gb200-max-tpt-megamoe (engine-identical to high-tpt; only prefill scale-out 2->3), so it reuses the same CONFIG_FILE. Validated standalone via e2e run 28711505502 at 8797 tok/s/GPU (median TTFT 58 ms) - above high-tpt's 8241 tok/s/GPU, since the extra prefill instance relieves the conc=4096 prefill bottleneck. Adds a matching perf-changelog entry under PR #2050. Signed-off-by: Ezra Silvera <ezra@il.ibm.com>
ezrasilvera
force-pushed
the
llm-d-initial
branch
from
3bd0b1d to
67be9f4
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28742470098 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28742470098 |
- llm-d/README.md: document endpoints.yaml as one entry per node (not per leader) to match server.sh add_role(); fix example ports (prefill 8200, decode 8000) and rewrite the rationale section (C2) - perf-changelog.yaml: canonicalize pr-link XXX -> 2050 (C4) Signed-off-by: Ezra Silvera <ezra@il.ibm.com>
ezrasilvera
force-pushed
the
llm-d-initial
branch
from
1672554 to
a5a267b
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28947792268 |
… GB200)
Add llm-d-vllm as an InferenceX multi-node framework and a DeepSeek-V4-Pro
FP4 GB200 disaggregated (P/D) benchmark curve.
Framework (file-discovery / no-k8s):
- benchmarks/multi_node/llm-d/{server.sh,submit.sh,job.slurm} + wrapper
dsv4_fp4_gb200_llm-d-vllm-disagg.sh; runner routing in
runners/launch_gb200-nv.sh (FRAMEWORK=llm-d-vllm, pyxis/enroot on GB200).
- Headless-TP follower support so pure-TP decode engines span multiple nodes
with a single leader endpoint + sidecar; EPP + Envoy front the P/D pool
(envoy.yaml, epp-config.yaml). Image build inputs under benchmarks/llm-d/.
- DeepSeek-V4 tokenizer support in utils/bench_serving/benchmark_serving.py;
multinode plumbing in benchmarks/benchmark_lib.sh.
Sweep curve (configs/nvidia-master.yaml + perf-changelog.yaml), 2 recipes:
- mid-curve 1P1D DEP8 (conc 256/512/1024), high-tpt 2P1D DEP8 (conc 4096),
max-tpt 3P1D DEP8 (conc 4096) share dsv4-fp4-gb200-mid-curve-megamoe.yaml.
- low-latency 1P1D pure-TP8 (conc 1) and low-middle 1P4D pure-TP8
(conc 256/512) share dsv4-fp4-gb200-low-latency.yaml.
Unique SLURM job name (RUNNER_NAME + GITHUB_RUN_ID/ATTEMPT) so a name-scoped
scancel from another job on the shared cluster cannot cancel this allocation
- fixes the max-tpt in-sweep kill (validated re-test run 28918650203,
8730 tok/s/GPU, matching standalone 8747 / reference 8797).
Signed-off-by: Ezra Silvera <ezra@il.ibm.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28964680257 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28964790948 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28965102413 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28965283789 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28965500067 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28966565621 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28968221515 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28972533160 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28974354671 |
2 participants
Adding llm-d into InferenceX Starting with DeepSeek-V4-Pro FP4 GB200 disaggregated benchmarks (mid-curve + high-tpt)
Summary
Adds llm-d (vLLM) as a new benchmark framework in InferenceX, with a no-k8s multi-node path (Envoy + EPP + P/D sidecar over SLURM/pyxis), and theDeepSeek-V4-Pro FP4 GB200 P/D-disaggregated recipes:
dsv4-fp4-gb200-llm-d-vllm-mid-curve-megamoe- 1P1D DEP8, conc 256/512/1024dsv4-fp4-gb200-llm-d-vllm-high-tpt-megamoe- 2P1D DEP8, conc 4096Both keys share one engine/EPP recipe (
benchmarks/multi_node/llm-d-recipes/dsv4-fp4-gb200-mid-curve-megamoe.yaml);topology and concurrency come from the master-config keys, not the recipe.
Notable
dsv4-fp4-gb200-dynamo-vllm- same SKU, model, ISL/OSL, disagg TP1/DP8/EP8 layout, and the same concurrency points (256/512/1024/4096).agg_*.jsonrows throughprocess_result.py(framework=llm-d-vllm, X=conc, Y=tput_per_gpu) and the same_gpus_/_ctx_/_gen_result-filename convention the multinode ingest already parses. The only downstream addition isthe framework label in InferenceX-app (separate PR).
Scope
Known merge-blocker
image:on both keys is a personal build (ghcr.io/ezrasilvera/llm-d-nokube-vllm:vllm0.23). It should be republished to the org registry and theimage:lines updated before merge.Related
llm-d-vllm-> "llm-d vLLM" inSemiAnalysisAI/InferenceX-app(FW_REGISTRY) so the new series is recognized and labeled on inferencex.semianalysis.com. https://github.com/SemiAnalysisAI/InferenceX-app/pull/524Before merge
image:on both keys.perf-changelog.yamlpr-linkfrompull/XXXto this PR's number.