Skip to content

perf: amortize per-sample startup and post comments on fork PRs#1506

Merged
JacksonWeber merged 2 commits into
microsoft:mainfrom
JacksonWeber:jacksonweber/perf-speedup-and-fork-pr-comments
Jun 16, 2026
Merged

perf: amortize per-sample startup and post comments on fork PRs#1506
JacksonWeber merged 2 commits into
microsoft:mainfrom
JacksonWeber:jacksonweber/perf-speedup-and-fork-pr-comments

Conversation

@JacksonWeber

Copy link
Copy Markdown
Contributor

What

Two perf-CI improvements, both motivated by the recent perf-regression job taking ~14 minutes per PR.

1. Make perf tests faster

Diagnosis from a recent run: 6 scenarios × 3 samples = 18 child Node processes, each paying ~14.7s of pure overhead (Node startup + importing applicationinsights' large dependency tree + useAzureMonitor() init) for only 6s of actual measurement.

Fix: refactor bench.mjs to accept --samples N and take all N samples back-to-back inside one child process (warmup runs once, then N timed durations). runBenchmarks.mjs now spawns 1 child per scenario instead of samples × scenarios. OpenTelemetry global-state isolation is preserved because each scenario still gets a fresh process.

Expected savings: ~3 min off each of the two Run X benchmarks steps.

2. PR comments on fork PRs

The previous inline sticky-pull-request-comment step was gated on head.repo.full_name == github.repository so fork PRs never received a comment.

Modeled on microsoft/opentelemetry-distro-python's split:

  • performance.yml now runs without pull-requests: write and uploads baseline.json, candidate.json, perf-comparison.md, pr-number.txt as the perf-results artifact.
  • performance-comment.yml (new) triggers via workflow_run after Performance completes, runs in the trusted base-repo context with pull-requests: write, downloads the artifact, validates the PR number is a positive integer, regenerates report.md from the JSON using the base branch's own trusted comparePerf.mjs (so attacker-supplied markdown is never posted under the writable token), and posts the sticky comment.

3. Docs

test/performanceTests/README.md updated to describe the workflow split (with security model) and the single-child-per-scenario speedup.

Verification

  • Locally verified bench.mjs + runBenchmarks.mjs end-to-end with --samples 2 --duration 1 --warmup 1 --scenarios OtelLogTest,OtelSpanTest: produces the expected JSON shape and comparePerf.mjs consumes it unchanged.
  • Both workflow YAML files parse cleanly.

Run all --samples of a given scenario inside a single child Node
process. OpenTelemetry global-state isolation is preserved because
each scenario still gets a fresh process, but Node startup, importing
applicationinsights' large dependency tree, and one-time SDK init
(useAzureMonitor()) are now paid once per scenario instead of once
per sample. On the perf CI runner this removes several minutes from
each "Run X benchmarks" step.

Split the perf workflow into producer/consumer to support fork PRs:
- performance.yml now runs without pull-requests:write and uploads
  baseline.json, candidate.json, perf-comparison.md, and pr-number.txt
  as the perf-results artifact. The inline sticky-comment step (which
  was gated on head.repo.full_name == github.repository, so forks
  never got comments) is removed.
- performance-comment.yml is new. It triggers on workflow_run after
  Performance completes, runs in the trusted base-repo context with
  pull-requests:write, downloads the artifact, validates the PR
  number is a positive integer, regenerates report.md from the JSON
  using the base branch's own trusted comparePerf.mjs (so attacker-
  supplied markdown is never posted under the writable token), and
  posts the sticky comment.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves performance CI by reducing benchmark runtime overhead and enabling performance comparison comments to be posted on fork-based pull requests in a safer, split-workflow model.

Changes:

  • Refactors the perf harness to take multiple samples per scenario within a single child process (bench.mjs + runBenchmarks.mjs) to amortize Node/module initialization costs.
  • Splits PR commenting into a new workflow_run workflow that runs in the base-repo context, downloads perf artifacts, regenerates markdown from JSON using trusted code, and posts a sticky PR comment.
  • Updates perf-test documentation to describe the workflow split and security model.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
test/performanceTests/runBenchmarks.mjs Passes --samples through to the bench child and aggregates per-sample ops/s into summary stats.
test/performanceTests/bench.mjs Collects N back-to-back timed samples in one process and outputs a samples[] JSON shape.
test/performanceTests/README.md Documents the CI workflow split and rationale/security model.
.github/workflows/performance.yml Removes PR write permissions and uploads perf outputs + PR number as an artifact for follow-on commenting.
.github/workflows/performance-comment.yml New workflow to download artifact in base context, regenerate report from JSON, and post sticky PR comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread test/performanceTests/bench.mjs Outdated
Comment thread .github/workflows/performance-comment.yml
Comment thread .github/workflows/performance-comment.yml Outdated
@JacksonWeber JacksonWeber requested a review from rads-1996 June 15, 2026 18:58
- bench.mjs: require Number.isInteger for --samples so 1.5 is rejected.
- performance-comment.yml: resolve PR number from workflow_run event payload (or listPullRequestsAssociatedWithCommit keyed off the untamperable head_sha) instead of the untrusted pr-number.txt artifact, so a malicious fork PR cannot trick the writable token into commenting on an unrelated PR.
- performance-comment.yml: drop the duplicated PERF_REGRESSION_THRESHOLD env var; comparePerf.mjs's built-in default of 15 now applies, removing the third source of truth.
- performance.yml: drop the now-unused pr-number.txt writer/upload.
@JacksonWeber JacksonWeber merged commit b441fad into microsoft:main Jun 16, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants