Skip to content

fix(memory): source forwarded env below memtrack's capability boundary#438

Closed
not-matthias wants to merge 1 commit into
mainfrom
cod-3076-mozilla-slack-message
Closed

fix(memory): source forwarded env below memtrack's capability boundary#438
not-matthias wants to merge 1 commit into
mainfrom
cod-3076-mozilla-slack-message

Conversation

@not-matthias

Copy link
Copy Markdown
Member

Context

Reported via Mozilla Slack thread (COD-3076): a neqo benchmark fails on the memory runner but succeeds on simulation. The benchmark panics in NSS init with:

called `Result::unwrap()` on an `Err` value: UnsupportedVersion

The workflow builds NSS 3.121 from source (the runner image ships 3.98) and points the benchmark at it via LD_LIBRARY_PATH. On the memory runner that variable never reached the benchmark, so it loaded the too-old system NSS.

Root cause

codspeed-memtrack holds file capabilities (cap_bpf, cap_perfmon, cap_sys_admin). A non-root user executing a capability binary enters glibc secure-execution mode (AT_SECURE=1), which strips LD_* variables from the environment before main().

The memory executor sourced the forwarded environment around memtrack:

bash -c "source ENV && memtrack track … '<bench>'"   ← LD_LIBRARY_PATH restored here
└─ codspeed-memtrack                                 ← cap binary: glibc strips LD_* again
   └─ bash -c <bench>                                ← inherits stripped env ❌

So LD_LIBRARY_PATH was restored and then stripped a second time, before the benchmark memtrack spawns could inherit it.

Walltime is unaffected because it layers privilege escalation (sudo/perf/systemd scope) outside the env re-source, keeping source at the innermost layer.

Fix

Source the env inside the command memtrack runs via its own bash -c, below the capability boundary:

codspeed-memtrack track … "source ENV && <bench>"
└─ codspeed-memtrack           ← strips LD_* (no longer matters)
   └─ bash -c "source ENV && <bench>"   ← re-sources below the cap binary
      └─ <bench>                        ← sees LD_LIBRARY_PATH ✅
  • Add prefix_command_with_env for the raw source <file> && <cmd> snippet (no outer bash -c) and reuse it from wrap_with_env.
  • MemoryExecutor::build_memtrack_command now prefixes the bench command and passes it as memtrack's argument; the outer bash wrapper is gone.

Test

test_memory_executor_forwards_ld_library_path (Linux + GITHUB_ACTIONS gated) exercises this end-to-end. The existing ..._forwards_path test never caught it because PATH is not an LD_* variable.

Note

If the memory test suite is ever run as root, memtrack's caps add nothing, secure-exec isn't triggered, and the test would pass even with the bug present, same limitation as the real-world scenario, which only affects the non-root runner user.

Verification

Reproduced the mechanism with a minimal capability binary that runs its argument via bash -c (memtrack's exact structure), as a non-root user:

nesting benchmark sees LD_LIBRARY_PATH
old (source around cap binary) empty ❌
new (source inside bench) /custom/lib

codspeed-memtrack holds file capabilities, so a non-root user executing it
enters glibc secure-execution mode (AT_SECURE=1), which strips LD_* variables
from the environment before main(). The memory executor sourced the forwarded
environment around memtrack, so LD_LIBRARY_PATH was restored and then stripped
again before the spawned benchmark could inherit it. Benchmarks relying on
LD_LIBRARY_PATH (e.g. neqo loading a locally built NSS) fell back to the system
library and panicked with UnsupportedVersion.

Source the env inside the command memtrack runs via `bash -c`, matching how
walltime already layers privilege escalation outside the env re-source. Add
prefix_command_with_env for the raw `source <file> && <cmd>` snippet and reuse
it from wrap_with_env.
@not-matthias not-matthias deleted the cod-3076-mozilla-slack-message branch July 7, 2026 09:17
@not-matthias not-matthias restored the cod-3076-mozilla-slack-message branch July 7, 2026 09:18
@greptile-apps

greptile-apps Bot commented Jul 7, 2026

Copy link
Copy Markdown

Greptile Summary

This PR moves memory-runner env forwarding below memtrack's capability boundary.

  • Adds a raw source <env-file> && <command> helper.
  • Passes the env-prefixed benchmark command into codspeed-memtrack track.
  • Adds a memory executor test for forwarded LD_LIBRARY_PATH.

Confidence Score: 4/5

The memory runner can lose the forwarded env file when memtrack runs the child shell as a different user.

  • The new placement fixes the reported LD_* stripping path.
  • The temp env file may be unreadable after memtrack's privilege drop.
  • The added test covers the non-root capability path, but not the root-to-runner user transition.

src/executor/memory/executor.rs

Important Files Changed

Filename Overview
src/executor/helpers/run_with_env.rs Adds prefix_command_with_env and keeps wrap_with_env as the shell-wrapped caller.
src/executor/memory/executor.rs Moves env sourcing into the command executed by memtrack's child shell.
src/executor/tests.rs Adds Linux memory-runner coverage for forwarding LD_LIBRARY_PATH.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
  participant Runner as codspeed runner
  participant EnvFile as temp env file
  participant Memtrack as codspeed-memtrack
  participant Bash as child bash -c
  participant Bench as benchmark

  Runner->>EnvFile: write forwarded environment
  Runner->>Memtrack: "track ... "source ENV && bench""
  Note over Memtrack: file capabilities strip LD_* before main
  Memtrack->>Bash: "bash -c "source ENV && bench""
  Bash->>EnvFile: source forwarded env
  Bash->>Bench: "run with restored LD_*"
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
  participant Runner as codspeed runner
  participant EnvFile as temp env file
  participant Memtrack as codspeed-memtrack
  participant Bash as child bash -c
  participant Bench as benchmark

  Runner->>EnvFile: write forwarded environment
  Runner->>Memtrack: "track ... "source ENV && bench""
  Note over Memtrack: file capabilities strip LD_* before main
  Memtrack->>Bash: "bash -c "source ENV && bench""
  Bash->>EnvFile: source forwarded env
  Bash->>Bench: "run with restored LD_*"
Loading

Fix All in Claude Code Fix All in Codex

Prompt To Fix All With AI
Fix the following 1 code review issue. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 1
src/executor/memory/executor.rs:62
**Privilege Drop Hides Env File**

When the runner starts memtrack as root and memtrack drops its child `bash -c` to `SUDO_UID`/`SUDO_GID`, the child now has to `source` a `NamedTempFile` created by the runner. That temp file is owner-only by default, so the benchmark shell can fail with `Permission denied` before restoring the forwarded environment; the old outer wrapper read the file before crossing into memtrack's dropped child process.

Reviews (1): Last reviewed commit: "fix(memory): source forwarded env below ..." | Re-trigger Greptile

// LD_* variables. Sourcing inside the benchmark shell restores them below
// that boundary.
let bench_command = get_bench_command(&execution_context.config)?;
let (bench_command, env_file) = prefix_command_with_env(&bench_command, &extra_env)?;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Privilege Drop Hides Env File

When the runner starts memtrack as root and memtrack drops its child bash -c to SUDO_UID/SUDO_GID, the child now has to source a NamedTempFile created by the runner. That temp file is owner-only by default, so the benchmark shell can fail with Permission denied before restoring the forwarded environment; the old outer wrapper read the file before crossing into memtrack's dropped child process.

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/executor/memory/executor.rs
Line: 62

Comment:
**Privilege Drop Hides Env File**

When the runner starts memtrack as root and memtrack drops its child `bash -c` to `SUDO_UID`/`SUDO_GID`, the child now has to `source` a `NamedTempFile` created by the runner. That temp file is owner-only by default, so the benchmark shell can fail with `Permission denied` before restoring the forwarded environment; the old outer wrapper read the file before crossing into memtrack's dropped child process.

How can I resolve this? If you propose a fix, please make it concise.

Fix in Claude Code Fix in Codex

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant