prime images transfer-bulk

[kcoopermiller]

Adds a bulk counterpart to prime images push --source-image: copy many existing registry images into prime in one command

prime images transfer-bulk --manifest transfers.jsonl
prime images transfer-bulk --harbor ./tasks
prime images transfer-bulk --hf R2E-Gym/R2E-Gym-Subset --column docker_image

• --manifest — JSONL, one transfer per line: {"source": "ghcr.io/org/app:v1", "image"?: "name:tag", "platform"?: "..."}.
• --harbor — transfers each task's prebuilt [environment] docker_image. This is the exact complement of push-bulk's Harbor mode (which skips those tasks), so the two commands together cover a full task directory.
• --hf — reads image references straight from a Hugging Face dataset column via the dataset-viewer API (no datasets dependency, no download). Takes --column and supports --hf-config/--hf-split, dataset URLs, and HF_TOKEN for gated datasets.

How it runs

1. Validate everything up front like push-bulk. --dry-run prints the resolved source → destination table without touching the API.
2. Submit one transfer per request through the sliding-window logic shared with push-bulk (run_bulk_jobs refactored out of run_bulk_push). The server's transfer limiter counts images/minute (10 personal / 60 team), so on a plain 429 the spec is requeued and submission pauses briefly while in-flight transfers keep getting polled. A quota 429 stops submission (rest marked SKIPPED); 20 consecutive deferrals with zero progress makes the run give up instead of pacing forever.
3. Poll to terminal status and write anything that didn't complete to a re-runnable failures manifest: prime images transfer-bulk --manifest transfer-bulk-failures.jsonl.

Verified end-to-end against R2E-Gym/R2E-Gym-Subset (4,578 images resolved via dry-run).

---

Note

Medium Risk
Large new CLI surface and shared bulk submission logic that talks to /images/build and external Hugging Face APIs; mistakes could mis-queue transfers or mishandle quota vs rate-limit 429s, but changes are client-side with extensive tests.

Overview
Adds prime images transfer-bulk to queue many copy-existing-image jobs into Prime (counterpart to single-image push --source-image and build-oriented push-bulk).

Input modes: JSONL manifest (source, optional image/platform), Harbor tasks using prebuilt [environment] docker_image (what push-bulk skips), and Hugging Face datasets via the dataset-viewer API (--hf, --column, optional config/split; HF_TOKEN for gated data). Up-front validation, --dry-run, and a re-runnable failures JSONL mirror push-bulk.

Shared bulk engine: run_bulk_push is refactored into generic run_bulk_jobs with an injectable submit callable. Transfers use it with SubmitRateLimited pacing on transfer 429s, quota aborts, and a cap on consecutive deferrals so a stuck rate limiter cannot hang the CLI forever.

^{Reviewed by Cursor Bugbot for commit 4ceac8f. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 38aa296. Configure here.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 38aa296de6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[kcoopermiller]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cb1fc1cdea

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[kcoopermiller]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 16c4977749

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[kcoopermiller]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 16c4977749

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".