Skip to content

feat/batches: upload large exec artifacts#1324

Draft
burmudar wants to merge 1 commit into
mainfrom
wb/artifact-upload
Draft

feat/batches: upload large exec artifacts#1324
burmudar wants to merge 1 commit into
mainfrom
wb/artifact-upload

Conversation

@burmudar
Copy link
Copy Markdown
Contributor

Batch executor jobs can produce stdout, stderr, and diffs large enough to bloat the final JSONL result stream. Upload oversized step artifacts from the existing src batch exec execution path instead of adding a separate command or user-facing auth flags.

src batch exec now discovers the executor job context from the environment, uses the existing executor job bearer token plus executor name auth model to stream large artifacts to the Batch Changes artifacts endpoint, and records returned references on AfterStepResult. Small outputs stay inline for compatibility, and per-step progress output is suppressed when artifact uploads are enabled so large logs are not duplicated into task-step JSONL events.

Related in sg/sg:

  • wb/proto-exec-upload
  • feat/batches: add get endpoint for executor artifacts

Test Plan:

  • go test ./cmd/src ./internal/batches/executor ./internal/batches/ui github.com/sourcegraph/sourcegraph/lib/batches/execution

Test plan

@burmudar burmudar self-assigned this May 25, 2026
Batch executor jobs can produce stdout, stderr, and diffs large enough to bloat the final JSONL result stream. Upload oversized step artifacts from the existing `src batch exec` execution path instead of adding a separate command or user-facing auth flags.

`src batch exec` now discovers the executor job context from the environment, uses the existing executor job bearer token plus executor name auth model to stream large artifacts to the Batch Changes artifacts endpoint, and records returned references on `AfterStepResult`. Small outputs stay inline for compatibility, and per-step progress output is suppressed when artifact uploads are enabled so large logs are not duplicated into task-step JSONL events.

companian branch in sg: wb/proto-exec-upload

Test Plan:

- go test ./cmd/src ./internal/batches/executor ./internal/batches/ui github.com/sourcegraph/sourcegraph/lib/batches/execution
@burmudar burmudar force-pushed the wb/artifact-upload branch from cae7f52 to f1430c1 Compare May 25, 2026 14:45
Copy link
Copy Markdown
Member

@keegancsmith keegancsmith left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should avoid wrapping the Body in your own reader which computes size. Check the docstring for NewRequestWithContext it already has special handling for a bunch of types you are using (mostly around setting ContentLength and retries).

If you wanna be fancy maybe something like

interface UploadBody {
  ContentLength() (int64, error)
  GetBody() (io.Reader, error)
}

would work nicely (check NewRequestWithContext implementation). Then just remember to wrap io.Reader in NopCloser and to seek to the start of the file on each call to GetBody (or just re-open the file?)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants