feat: remote training over SSH (no HF Jobs/account required)

[ZouzouWP]

The only non-local training option was HF Jobs — cloud, paid, and tied to a Hugging Face account. This adds a way to train on any machine you can already SSH into (a spare GPU box, a rented cloud instance), with no account or extra service involved.

What changed

• New ssh_remote job target, alongside local and hf_cloud. The connection form asks for host/port/username, an optional private key path (falls back to the local ssh-agent/default identity — LeLab never handles or stores a password), a remote working directory, and a free-form "remote Python command" (e.g. python, or source ~/venv/bin/activate && python), since LeLab has no way to know how the remote server is set up.
• New lelab/runners/ssh_remote.py (SshRemoteJobRunner): stages the dataset via scp, launches training over ssh, and tracks the local ssh client's PID as process_pid.
• Checkpoints are pulled back with a rate-limited scp (at most once every 30s, since the frontend polls checkpoints roughly every 5s) into the same local output_dir a local job would use — so checkpoint listing and inference reuse the exact same local code path, no separate "remote checkpoint" representation needed anywhere.

sequenceDiagram
    participant U as User
    participant T as TargetCard (Training page)
    participant API as LeLab backend
    participant R as SshRemoteJobRunner
    participant S as Remote server

    U->>T: Fill host/user/key/workdir/python cmd, start training
    T->>API: POST /jobs/training {target: {runner: ssh_remote, ssh: {...}}}
    API->>R: start()
    R->>S: scp dataset -> remote_workdir
    R->>S: ssh: run training command
    loop every ~30s at most (rate-limited)
        API->>S: scp checkpoints/ -> local output_dir
    end
    API-->>T: same checkpoint/metrics UI as a local job

Loading

Testing

Implemented and exercised locally (dataset staging, command construction, checkpoint-pull rate limiting), but not yet run end-to-end against a real remote host — needs a pass against a reachable GPU box before merge.