[AMD] add dsv4 sglang disagg / 新增 DSv4 SGLang 分离式基准测试#1818
[AMD] add dsv4 sglang disagg / 新增 DSv4 SGLang 分离式基准测试#1818billishyahao wants to merge 36 commits into
Conversation
billishyahao
requested review from
1am9trash,
seungrokj and
yctseng0211
as code owners
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit c22652b. Configure here.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27731746053 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27731746053 |
functionstackx
added
the
all-evals
# Conflicts: # perf-changelog.yaml
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27731746053 |
| description: | ||
| - "init submission of dsv4 sglang disagg " |
There was a problem hiding this comment.
can u also ur ai agent include descriptions + links of some of the bug fix PRs in here like
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27893589025 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27896968169 |
|
hi @billishyahao there seems to be an accuracy issues with TP8+TP8. codex has narrowed it down to conc=4, here is the bug report for when u wake up, please take a look https://github.com/SemiAnalysisAI/InferenceX/actions/runs/27896968169/job/82550287079?pr=1818 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28693213723 |
functionstackx
changed the title
…-f backstop Multi-node disagg CI hung after benchmarks completed: server_sglang.sh printed 'Script completed successfully' on both nodes, but the container never exited (srun/CI step hung ~6h until cancelled). Root cause: teardown killed only the server's main PID. sglang launch_server is multi-process (python + TP-scheduler children); surviving children kept the process-substitution tee's pipe open, so the container's outer '| tee' never got EOF and 'exec docker run' never returned. This was introduced by the process-sub change in 3c19127 (kill target moved from tee to main PID). Verified locally: image-independent (rocm/sgl-dev and lmsysorg builds behave identically); reproduced hang with 'kill main-pid' and clean exit with setsid + process-group kill. Fix (server_sglang.sh): launch prefill0/prefill/decode under setsid so each server tree sits in its own process group, capture the pgid, and teardown with 'kill -TERM -$pgid' to reap the whole tree (mirrors the router launch). Backstop (job.slurm): restore the cleanup-trap 'srun docker rm -f $DOCKER_CONT_NAME' (ported from InferenceY 51ebfa88) so any residual hang still lets the job tear down its container on scancel/timeout.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28696443568 |
2 similar comments
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28696443568 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28696443568 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28876195758 |
billishyahao
added
evals-only
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28876603659 |
billishyahao
added
all-evals
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28879220277 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28879220277 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28885560673 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28885560673 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28911918964 |
4 participants
cc @Duyi-Wang
Note
Medium Risk
Touches shared disagg launch paths (
server_sglang.sh,models.yaml) for all models, not only DSv4; behavior changes when EP is disabled and MoE auto-sizing is partially commented out.Overview
Adds
dsv4-fp4-mi355x-sglang-disaggto the AMD master benchmark matrix (8k/1k, non-MTP) with sweeps over pure TP8, DEP8 (MoRI KV + MoE a2a), and dp-attention + TP-MoE, plus a new workflow runnerdsv4_fp4_mi355x_sglang-disagg.shand a perf-changelog entry.The multi-node harness is extended for DSv4 PD: a
DeepSeek-V4-Problock inmodels.yaml(dsv4 attention backend, mori disagg, prefilldisable_cuda_graph) and matching MoRI/kernel env overrides inenv.sh; the bench client uses--dsv4framing instead of chat templates.server_sglang.sh/models.yamlrefactor MoE CLI soep_flags(mori a2a, deepep, fake dispatch) apply only when EP is on—ep=1stays TP-MoE even with dp-attention—and prefill can honor per-modeldisable_cuda_graph,context_length, and optionalMORI_NUM_MAX_DISPATCH_TOKENS_PER_RANK_*overrides.submit.shthreadsDRY_RUNfor previewing composed launch commands on a real allocation.Reviewed by Cursor Bugbot for commit f56f8de. Bugbot is set up for automated code reviews on this repo. Configure here.
中文说明
在 AMD 主基准测试矩阵中新增
dsv4-fp4-mi355x-sglang-disagg配置（8k/1k，非 MTP），覆盖纯 TP8、DEP8（MoRI KV + MoE a2a）及 dp-attention + TP-MoE 多种扫描点。新增工作流运行器脚本dsv4_fp4_mi355x_sglang-disagg.sh和 perf-changelog 条目。扩展多节点 harness 以支持 DSv4 的 EP 和 MoE 配置。