Skip to content

MILAB-6382: dual-mode operation via exec.hasGpu - #1

Draft
blackcat wants to merge 1 commit into
mainfrom
feat/MILAB-6382-gpu-support
Draft

MILAB-6382: dual-mode operation via exec.hasGpu#1
blackcat wants to merge 1 commit into
mainfrom
feat/MILAB-6382-gpu-support

Conversation

@blackcat

@blackcat blackcat commented Jun 4, 2026

Copy link
Copy Markdown
Contributor

Summary

gpu-test now runs cleanly on both GPU-enabled and CPU-only clusters. The workflow gates .gpuMemory() on exec.hasGpu and passes a --mode {gpu,cpu} flag to the software so the Python side runs the matching path with no runtime detection. The CPU branch is first-class — the block returns a "no GPU available" report and exits successfully, never errors.

First of six per-block PRs in the GPU-on-GCP plan — pattern reference for the remaining blocks (clonotype-space, immuno-match, 3d-structure-prediction, paratope-clustering, compositional-analysis).

Why this matters

Calling .gpuMemory() against a backend that reports exec.hasGpu = false is a permanent runner error (SDK changelog: "Calling .gpuMemory() against a backend that reports no GPU now produces a permanent error from the runner driver"). Today's gpu-test workflow calls .gpuMemory() unconditionally when args.gpuMemory is set — fine on a GPU cluster, broken on a CPU-only one. This PR fixes the gate AND moves the GPU-vs-CPU decision out of Python runtime detection into tengo, where it belongs (per the project-wide GPU plan).

Changes

File What
workflow/src/main.tpl.tengo Wrap .gpuMemory() in if exec.hasGpu; always pass --mode gpu / --mode cpu so the software path matches
software/gpu-info/src/main.py Add --mode {gpu,cpu}. CPU mode skips probes and emits a minimal report; GPU mode runs the existing flow unchanged. format_report() shows the active mode and refines the SUMMARY for CPU mode
model/src/index.ts GpuReport.mode: 'gpu' | 'cpu'
ui/src/pages/GpuInfoPage.vue Summary banner shows "CPU branch (exec.hasGpu = false)" alongside "GPU NOT AVAILABLE" when running on CPU
.changeset/milab-6382-gpu-support.md Minor bump across workflow / gpu-info / model / ui

Existing user-controllable args (cpu, mem, gpuMemory, matrixSize, seed) keep working — gpuMemory is just ignored on the CPU branch.

Test plan

  • python3 main.py --mode cpu --seed 42 → fast, no probes, SUMMARY: GPU NOT AVAILABLE (CPU branch, no probes attempted)
  • python3 main.py --mode gpu --seed 42 → existing detection runs; on this dev host (no GPU) reports SUMMARY: GPU NOT AVAILABLE as before
  • pnpm type-check (turbo) → 8/8 packages OK
  • End-to-end on the GPU staging cluster after PR 1 (https://github.com/milaboratory/pl/pull/1934) deploys → pod lands on gpu-l4-24g, banner shows GPU AVAILABLE, benchmark reports speedup > 1
  • End-to-end on a CPU-only cluster (or any cluster with PL_RUNNER_GPU_AVAILABLE=disabled) → block returns successfully, banner shows GPU NOT AVAILABLE + CPU branch (exec.hasGpu = false)

Pattern for follow-up block PRs

This block was the simplest case (already-passing gpuMemory arg, the Python diagnostic was easy to split into two branches). The remaining blocks need the same shape — workflow gate + explicit software flag + remove runtime detection. The harder ones:

  • clonotype-space — switch the cuML auto backend to explicit cuml / sklearn per branch (no more try: import cuml); drop the fallback.
  • 3d-structure-prediction — add a CUDA Dockerfile variant + --device cuda|cpu flag (currently CPU-pinned at the Dockerfile level).
  • paratope-clustering — drop map_location="cpu", take device from --device.
  • compositional-analysis — TF auto-detect → explicit CUDA_VISIBLE_DEVICES="" in CPU mode.
  • immuno-match — HuggingFace Trainer no_cuda from --device.

🤖 Generated with Claude Code

The block now runs cleanly on both GPU and CPU-only clusters — calling
.gpuMemory() on a backend without GPU is a permanent runner error, so
the workflow has to gate on exec.hasGpu. The same gate decides what to
tell the software: 'gpu' mode runs the existing detection + benchmark;
'cpu' mode skips the probes and emits a fast "no GPU available" report.
gpu-test never errors when GPU is absent — the CPU branch is a
first-class behaviour, not a fallback.

Workflow (main.tpl.tengo): wrap .gpuMemory() in `if exec.hasGpu`. Pass
the new --mode flag (gpu/cpu) so the software branch matches what the
workflow asked for. No runtime detection in Python anymore.

Software (gpu-info/src/main.py): add --mode {gpu,cpu}. CPU mode skips
the CuPy / PyTorch / nvidia-smi probes and returns a minimal but
schema-compatible report with gpu_available=false and a benchmark
'skipped' message. GPU mode runs the existing flow unchanged.
format_report() shows the active mode in the header and refines the
SUMMARY line for CPU mode.

Model: GpuReport gains a `mode: 'gpu' | 'cpu'` field. Existing fields
untouched — the CPU-mode JSON still satisfies the schema (empty/zero
probes, skipped benchmark).

UI (GpuInfoPage.vue): when mode === 'cpu', the summary banner shows a
small "CPU branch (exec.hasGpu = false)" label alongside the existing
"GPU NOT AVAILABLE" headline so users understand it ran the CPU path on
purpose, not as a failure.

Changeset: minor bump across workflow / gpu-info / model / ui.

Smoke test:
  python3 main.py --mode cpu --seed 42  → fast, no probes, SUMMARY: GPU
    NOT AVAILABLE (CPU branch, no probes attempted)
  python3 main.py --mode gpu --seed 42  → existing detection runs;
    on a non-GPU host reports SUMMARY: GPU NOT AVAILABLE as before
  pnpm type-check (turbo) — 8/8 packages OK

Reference: docs/text/work/ad-hoc/gcloud-gpu-plan.md PR 3 (gpu-test row).
First per-block PR — pattern reference for the remaining five blocks.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a dual-mode operation (GPU vs. CPU) for the GPU test workflow. The execution path is decided in the Tengo workflow using the exec.hasGpu gate, which passes a --mode flag to the Python backend. In CPU mode, the backend skips GPU probes and benchmarks to return a fast, schema-compatible report, and the UI is updated to display a CPU branch label. Feedback on the changes suggests adding the missing torch_cuda field to the TypeScript GpuReport interface to fully align with the backend's report schema.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread model/src/index.ts
// software ran the full detection + benchmark.
// 'cpu' = workflow ran on the CPU branch; probes were skipped and the
// block returned a fast "no GPU available" report without erroring.
mode: 'gpu' | 'cpu';

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The torch_cuda field is populated by the Python backend (both in gpu and cpu modes) but is currently missing from the TypeScript GpuReport interface. Adding it ensures the TypeScript model fully and accurately represents the backend report schema.

Suggested change
mode: 'gpu' | 'cpu';
mode: 'gpu' | 'cpu';
torch_cuda: {
available: boolean;
device_count: number;
cuda_version: string | null;
cudnn_version: string | null;
devices: Array<{
index: number;
name: string;
total_memory_mb: number;
free_memory_mb?: number;
major: number;
minor: number;
multi_processor_count: number;
}>;
error?: string;
};

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants