Skip to content

base images with -torch missing system dependency #302

Description

@keighrim

The *-torch2 base images (torch2, ffmpeg-torch2, opencv4-torch2, and everything downstream including all *-hf5) ship PyTorch but no C toolchain.

As the torch team is progressively replacing hand-written per-architecture CUDA kernels with Triton-generated ones (more portable across GPUs, easier to maintain), recent torch (2.10, Jan 2026) routes some eager ops (those run op-by-op in PyTorch's default mode, i.e. not traced/compiled under torch.compile) through Triton kernels (via the torch._native op registry), and Triton compiles a small host-side CUDA driver shim at runtime (this step needs a C compiler + libc headers).
On the slim python:3.10-slim base those are absent, so GPU inference dies at the first model.generate():

RuntimeError: Failed to find C compiler ...
# and after adding gcc alone:
.../triton/backends/nvidia/include/cuda.h: fatal error: stdlib.h: No such file or directory

This can hit any torch-based app, e.g. the SmolVLM2 captioner.
The recent release of the smol2 app had to add these system packages to its own Containerfile; this issue proposes the same mitigation at the base-image level so every torch-based app gets it for free.

Mitigation (proposed)

Add the minimal toolchain to the three *-torch2 containerfiles, where torch is installed:

RUN apt-get update \
    && apt-get install -y --no-install-recommends gcc libc6-dev \
    && rm -rf /var/lib/apt/lists/*

gcc + libc6-dev is sufficient (verified); build-essential installs the full C toolchain but is not needed.

Future direction

This is a workaround, not a cure: we ship a runtime build toolchain in a slim inference image only because Triton compiles kernels on-device.
The PyTorch 2.13 notes, released last week, point the same way:

  • PyTorch is adding more runtime codegen backends (CuTeDSL alongside Triton), so the "kernels compile on the user's machine" direction is deepening, not reversing -- but that work is Inductor / torch.compile only, so it does not touch our eager path.
  • No change yet to Triton's runtime-compiled driver shim (its own # TODO: make static), so the compiler requirement persists.
  • CUDA 13.0 is now the default build -- a separate driver-floor consideration for whatever CUDA version future base images track.

Another mitigation is to pin torch (<=2.9.*) so Triton is not required at all.
That is not desirable, though: pinning down would freeze the base images on an old torch and forgo the bug fixes, performance work, and transformers/CUDA compatibility the torch team ships in newer releases.
Ideally, upstream would ship a pre-compiled driver shim and pip install would resolve every dependency -- but until that happens, keeping a minimal C toolchain in the base is the best mitigation, IMHO.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

Status
Todo

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions