[AMD] DSv4 FP4 MI355X agentX vLLM #2109
Conversation
Pin LMCache to a specific commit to work around vLLM scheduler ValueError with hybrid KV cache manager. Bump vLLM image to nightly, tune LMCache config (blocking timeout, transfer mode), and narrow sweep to lmcache conc=72 for focused debugging. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
seungrokj
requested review from
1am9trash,
billishyahao,
chunfangamd and
yctseng0211
as code owners
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
seungrokj
changed the title
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| trap cleanup_lmcache_server EXIT | ||
|
|
||
| cleanup_agentic_services() { | ||
| local exit_code=$? | ||
| trap - EXIT INT TERM | ||
| set +e | ||
| stop_background_process_tree "$ROUTER_PID" "vLLM router" | ||
| stop_background_process_tree "$SERVER_PID" "vLLM server" 60 | ||
| stop_background_process_tree "$MOONCAKE_MASTER_PID" "Mooncake master" | ||
| exit "$exit_code" | ||
| } | ||
| trap cleanup_agentic_services EXIT | ||
| trap 'exit 130' INT | ||
| trap 'exit 143' TERM | ||
|
|
||
| wait_for_lmcache_ready() { | ||
| { set +x; } 2>/dev/null | ||
| local attempts="${LMCACHE_READY_ATTEMPTS:-120}" | ||
| local tail_pid="" | ||
|
|
There was a problem hiding this comment.
🟡 The lmcache branch installs two EXIT traps sequentially at lines 227 and 238 — bash only keeps the last handler per signal, so cleanup_agentic_services silently replaces cleanup_lmcache_server and (because it starts with trap - EXIT INT TERM and never touches $LMCACHE_PID) leaves the LMCache MP server orphaned on every exit path. The server holds LMCACHE_L1_SIZE_GB (~2.5 TB) of pinned host DRAM and stays bound to ports 5555/8080, which can block same-node retries. Fix by calling cleanup_lmcache_server from cleanup_agentic_services (or add stop_background_process_tree "$LMCACHE_PID" "LMCache server"), matching the single-trap pattern in kimik2.5_fp4_b200.sh:54.
Extended reasoning...
The bug\n\nIn the lmcache) branch of dsv4_fp4_mi355x_vllm.sh, two trap ... EXIT statements are registered in sequence:\n\nbash\n# line 227\ntrap cleanup_lmcache_server EXIT\n...\n# line 238\ntrap cleanup_agentic_services EXIT\ntrap 'exit 130' INT\ntrap 'exit 143' TERM\n\n\nBash only keeps one handler per signal — the second trap call silently overwrites the first. The replacement handler cleanup_agentic_services explicitly clears any residual handler with trap - EXIT INT TERM at its top, then only stops ROUTER_PID, SERVER_PID, and MOONCAKE_MASTER_PID. It never references LMCACHE_PID and never calls cleanup_lmcache_server. The MOONCAKE_MASTER_PID cleanup is also dead in this branch — that variable is only assigned in the mooncake) branch.\n\n### Why stop_background_process_tree $SERVER_PID does not reach it\n\nThe LMCache server is launched at line 341-342 as "${LMCACHE_CMD[@]}" > "$LMCACHE_LOG" 2>&1 &, so it is a sibling of the vLLM SERVER_PID under the main script's process group, not a descendant. stop_background_process_tree (benchmarks/benchmark_lib.sh:190) kills the given PID plus its descendants — walking down from SERVER_PID cannot reach a sibling.\n\n### Reference pattern\n\nbenchmarks/single_node/agentic/kimik2.5_fp4_b200.sh:54 uses only trap cleanup_lmcache_server EXIT with no override, which is what the author of this new script almost certainly intended before copy-pasting the mooncake-branch cleanup_agentic_services block below the lmcache-branch trap.\n\n### Step-by-step proof\n\n1. KV_OFFLOAD_BACKEND=lmcache, we enter the lmcache) branch.\n2. Line 227 registers trap cleanup_lmcache_server EXIT.\n3. Line 238 executes trap cleanup_agentic_services EXIT — the EXIT handler slot now points to cleanup_agentic_services; cleanup_lmcache_server is defined but unreachable via signal.\n4. LMCache server is spawned at line 341-342; LMCACHE_PID=$! captured.\n5. Script runs, then exits (normal, error, or SIGTERM → exit 143 → EXIT handler fires).\n6. cleanup_agentic_services runs: does trap - EXIT INT TERM; stops $ROUTER_PID, $SERVER_PID, $MOONCAKE_MASTER_PID (unset in this branch, no-op); exits.\n7. $LMCACHE_PID still alive — the process is not a child of SERVER_PID, and no code path kills it. It continues to hold LMCACHE_L1_SIZE_GB (TOTAL_CPU_DRAM_GB / (8 / TP) ≈ 2.5 TB with TP=8) of pinned host memory and to bind LMCACHE_PORT=5555 + LMCACHE_HTTP_PORT=8080.\n\n### Impact under the current sweep\n\nThe current amd-master.yaml sweep is narrowed to a single point (conc-list: [72]), so there is no in-sweep same-container retry. Under slurm, job-scoped cgroup teardown will reap the orphan at job boundary, bounding the practical damage. Any local re-run inside the same container/pod, uncommented sweep expansion back to multiple conc values, or interactive debugging on the runner will collide on ports 5555/8080 and/or fail cudaHostAlloc from leftover pinned pages.\n\n### Fix\n\nAny one of:\n\n1. Invoke cleanup_lmcache_server from inside cleanup_agentic_services (recommended — keeps the composed handler).\n2. Add stop_background_process_tree "$LMCACHE_PID" "LMCache server" to cleanup_agentic_services.\n3. Drop the second trap cleanup_agentic_services EXIT and let the single cleanup_lmcache_server handler stand (matches kimik2.5_fp4_b200.sh).
| git clone https://github.com/kvcache-ai/Mooncake.git | ||
| cd Mooncake | ||
| bash dependencies.sh | ||
| mkdir build | ||
| cd build | ||
| cmake .. | ||
| make -j | ||
| sudo make install # optional, make it ready to be used by vLLM/SGLang | ||
| cd .. | ||
| cd .. | ||
|
|
||
| python3 -c "from mooncake.store import MooncakeDistributedStore" >/dev/null | ||
| export INFERENCEX_MOONCAKE_MAX_TRANSFER_BATCH_KEYS=32 | ||
| python3 "$(dirname "$0")/patch_vllm_mooncake_transfer_batches.py" |
There was a problem hiding this comment.
hi @seungrokj whats the reason for needing to build mooncake from source? i thought mooncake has prebuilt ROCm wheels
pip install mooncake-transfer-engine-non-cuda
https://kvcache-ai.github.io/Mooncake/getting_started/quick-start.html
+viz @cquil11
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28840175467 |
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28843511906 |
Apply vllm-project/vllm#45497 scheduler fix at runtime via gist to resolve ValueError in _update_requests_with_invalid_blocks with hybrid KV cache manager. Expand lmcache conc-list to [16, 32, 48, 64]. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28850302356 |
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add DP-attention conc=[40,56,72] sweep, narrow lmcache to conc=[32,40], and reduce dram-utilization from 0.80 to 0.60. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28856284153 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28861274175 |
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28869263846 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28869505738 |
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28910441198 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28911223583 |
2 similar comments
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28911223583 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28911223583 |
|
/reuse-sweep-run |
seungrokj
changed the title
| # MI355X nodes have ~2.7 TiB of host DRAM available for offload; | ||
| # reserve 2.5 TB for the offload pool (leaves ~200 GB headroom for | ||
| # worker RSS / page cache / slurm cgroup). | ||
| TOTAL_CPU_DRAM_PARTITION_GB="$((TOTAL_CPU_DRAM_GB / (8 / TP)))" |
There was a problem hiding this comment.
This will not be needed anymore. will not hurt the CPU DRAM size for TP8 but will remove it in the next PR.
|
|
||
| python3 -c "import lmcache.integration.vllm.lmcache_mp_connector" >/dev/null | ||
|
|
||
| TOTAL_CPU_DRAM_PARTITION_GB="$((TOTAL_CPU_DRAM_GB / (8 / TP)))" |
There was a problem hiding this comment.
This will not be needed anymore. will not hurt the CPU DRAM size for TP8 but will remove it in the next PR.
2 participants
Summary
1720917eto work around vLLM schedulerValueErrorin the hybrid KV cache manager's_update_requests_with_invalid_blockspath09663abde) for DSv4-Pro MI355X agentic configlmcache_driventransfer mode, add--supported-transfer-modeflagTest plan
EngineDeadErrororValueErrorin scheduler🤖 Generated with Claude Code