feat: optional AWS Lambda MicroVM stateful sandbox backend#16
Open
danny-avila wants to merge 44 commits into
Open
feat: optional AWS Lambda MicroVM stateful sandbox backend#16danny-avila wants to merge 44 commits into
danny-avila wants to merge 44 commits into
Conversation
Moves the sandbox execute POST from workers.ts into a pluggable HttpSandboxBackend behind getSandboxBackend(). Adds CODEAPI_SANDBOX_BACKEND config (http default) and a startup policy check that rejects lambda-microvm until that backend lands. No wire behavior change: the signed request body and headers pass through byte-identical, and axios errors are rethrown untouched so the worker's abort/timeout/sandbox-error mapping is unchanged.
… launch) Adds runtime_session_hint to /exec (validated, optional) and derives rt_<sha256(namespace,user,hint)> server-side so a client hint can never collide across tenants. The id rides JobData into the sandbox backend context; stateless mode (default) derives nothing and enqueues byte-identical job data. The registry maps runtime_session_id -> MicroVM record in Redis with the replay-state lock discipline: SET NX PX mutex, CAS-delete release, token-fenced record writes/removals, monotonic generation counter for launch fencing, and a last-seen zset for the idle sweeper. No consumers yet - the Lambda backend lands behind CODEAPI_SANDBOX_BACKEND.
Adds the LambdaMicrovmClient interface plus AwsLambdaMicrovmClient (@aws-sdk/client-lambda-microvms, isolated to lambda-client-aws.ts and absent from http-only bundles), a transport-free in-memory fake for bun tests, and Redis-backed per-second token buckets with poison backoff for the account-wide control-plane TPS limits. Mapping notes from the SDK typings: RunMicrovm takes imageIdentifier + imageVersion, connector ARN arrays, native idlePolicy (auto-suspend / auto-terminate / auto-resume), runHookPayload, and a clientToken idempotency key; auth tokens come back as an X-aws-proxy-auth header map with minute-granularity expiry (max 60).
…kend
Adds the AWS Lambda MicroVM execution path (opt-in, config-gated behind
CODEAPI_SANDBOX_BACKEND=lambda-microvm; default http is unchanged):
- api/Dockerfile lambda-microvm-runner target: the existing sandbox-runner
packaged for Lambda MicroVMs (arm64, port 8080, /pkgs baked, no libkrun
launcher since the MicroVM is the VM boundary).
- api lifecycle hooks at /aws/lambda-microvms/runtime/v1/{ready,run,resume,
suspend,terminate}: no-op acks in Phase 1-2, /run captures the per-VM
runHookPayload (Phase 3 checkpoint attachment point).
- LambdaMicrovmSandboxBackend (stateless mode): run -> poll RUNNING ->
mint X-aws-proxy-auth token -> health -> proxy /api/v2/execute ->
terminate (incl. terminate-on-abort); throttle-aware, metrics-wired.
- Startup policy rules (blocking-PTC reject, image-ARN required, hardened
egress-connector required, token-TTL cap, no shell in prod).
- entrypoint raises RLIMIT_NOFILE hard cap to 65536: the AL2023 MicroVM
base caps it at 1024 (below the 2048 sandbox default), which made every
in-guest nsjail job fail setrlimit with EPERM. Verified on live AWS.
Live-verified on AWS Lambda MicroVMs (us-east-1): image builds and
snapshots, nsjail runs with additionalOsCapabilities ALL, bash+python
execute (code 0), and suspend/resume preserves process+memory state
with ~1.2s auto-resume-on-traffic.
Turns the semi-stateless runner stateful when a VM is bound to a runtime
session: instead of a fresh /mnt/data workspace per /execute, the VM
reuses ONE persistent workspace and one pinned UID across calls, so
files, installed packages, and chDB dirs survive between tool calls.
Modular and off by default: gated by TWO independent locks, so the
legacy fresh-per-job path is byte-identical when either is absent.
1. Image-level SANDBOX_SESSION_WORKSPACE_ENABLED — set only in the
lambda-microvm-runner target; the K8s image cannot enter session
mode regardless of any payload.
2. Per-launch /run runHookPayload {runtime_session_id, session_workspace}
— the control plane opts a specific VM in.
Mechanism:
- session-workspace.ts: process singleton bound by the /run hook,
unbound by /terminate; holds the pinned UID + output-diff and
priming-dedup state.
- workspace-isolation.ts: ensureSessionWorkspace (stable id, contents
preserved, reaper-protected) + resetSessionWorkspace teardown.
- Job branches only at three seams — prime (reuse workspace+UID, skip
re-downloading unchanged inputs), walkDir file surfacing (skip prior
outputs unchanged by size+mtime — output diffing), cleanup (keep
workspace+UID for the session).
Verified end-to-end in the arm64 runner container: fresh mode wipes
between calls; session mode persists files across calls (incl. across
languages) and accumulates; /terminate wipes and rebind gives a clean
slate. 274 api tests pass (fresh path unchanged).
Connects the proven runner session workspace to the product: when a
runtime session id is present and the mode is affinity/strict, the
Lambda backend finds-or-launches ONE warm VM per runtime_session_id via
the registry, delivers the /run runHookPayload
{runtime_session_id, session_workspace:true} that activates the runner's
persistent workspace, and reuses that VM across executes. AWS idlePolicy
(autoResume) suspends the VM when idle and wakes it on the next request,
so there is no explicit resume in the hot path.
- Serializes per session on the registry lock; strict contention -> 409
(publicExecutionFailure maps RUNTIME_SESSION_BUSY), affinity contention
-> correct stateless one-shot fallback (files always ride the payload).
- Generation-fenced launch: a fenced worker terminates its orphan VM.
- Stateless path unchanged (one VM per exec, terminate after).
- Lifts the stateless-only startup policy; adds session/fallback/lock
metrics. 345 service tests pass, incl. reuse/serialize/fallback/fence.
… across expiry
Makes an expiring/evicted MicroVM's state survive a relaunch — the
difference between real statefulness and just re-implementing the
existing file-ref system. The file-ref path only brings back files
surfaced as CodeEnvRefs; checkpoint/restore brings back the WHOLE
workspace: pip-installed packages, venvs, chDB dirs, caches, and files
with unsupported extensions.
Two runner endpoints (session-mode only, 409 otherwise so the legacy
runner exposes nothing new):
- GET /api/v2/session/checkpoint streams tar.gz of the workspace
- POST /api/v2/session/restore replaces the workspace from one,
re-owned to the session's pinned UID
Control-plane driven: the orchestrator pulls the checkpoint over the
authed proxy and owns the S3 write, so the untrusted VM never gets S3
credentials (matches the report's checkpoint-capability security model).
Verified end-to-end with two containers simulating VM expiry: VM1 builds
a python module tree + a 2KB unsupported-extension binary, is
terminated; a fresh VM2 shows the state absent (all file-refs give you),
then after restore imports the module (greet()=42) and reads the binary
(2048 bytes) — full workspace continuity across a VM swap. 276 api
tests pass.
Closes the 8h-rollover / eviction story so perceived statefulness is
automatic instead of control-plane-by-hand. After each successful
session exec (lock still held), pull the workspace tar from the warm VM
and store it to S3 under a deterministic key (rtsx-checkpoints/<id>.tar.gz)
so recovery survives even registry loss; record the pointer under the
same fenced write. On relaunch, a fresh session VM restores its
predecessor's checkpoint before the first exec, making an expired/evicted
VM invisible.
- Coverage is complete and tear-free: the workspace only mutates during
an exec and execs serialize on the session lock, so the post-exec
checkpoint always captures the latest state; a busy lock means a newer
exec will checkpoint instead.
- Never fatal: a missed checkpoint degrades to file-ref recovery, a
failed restore to a fresh workspace ('relaunched must be correct').
- Off => warm reuse still works, cross-VM restore falls back to file
refs. CheckpointStore injected (Minio prod / Memory in tests); byte
cap + timeout bound the transfer; checkpoint/restore/bytes metrics.
354 service tests pass incl. checkpoint-after-exec, restore-before-first
-exec ordering, no-restore-on-reuse, disabled skip, failure non-fatal.
…eader Deliver session mode per /execute request instead of the /run lifecycle hook. Lambda image build hooks only route on the snapshot-compatible base container image, and enabling any runtime hook forces the /ready build hook (which never reaches a stock container listener), so hookless image builds are the reliable path. The runner binds its persistent workspace from the header; the backend stamps it on the proxied execute in session mode and drops runHookPayload from RunMicrovm. Verified on a real hookless MicroVM.
|
|
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
…lper Operator guide for the optional AWS Lambda MicroVM backend: the cross-repo picture, from-scratch AWS setup, a full config reference, operating modes, alternative AWS methods (base image, checkpoint store, egress, quota), the PTC replay/blocking distinction, and the hard-won runbook gotchas. - docs/lambda-microvm/terraform: prerequisites module (checkpoint + artifact buckets, build + logging-only execution roles with the sts:TagSession / logs:* trust the build needs, CloudWatch log groups, checkpoint access policy). terraform validate + fmt clean. - service/scripts/create-microvm-image.ts: guaranteed-correct hookless CreateMicrovmImage helper (ALL caps + cgroupv2 off baked in), the one provisioning step Terraform can't own.
…, +)
Fixes from the Macroscope review pass:
- registry: derive RUNTIME_SESSION_LOCK_TTL_MS from the actual launch/health/
execute/checkpoint budgets (was a 60s placeholder, could expire mid-work at
defaults); guard readRuntimeSessionRecord JSON.parse so a corrupt key reads
as missing instead of wedging the session.
- checkpoint: fence (fenced record write) BEFORE the deterministic-key S3 put
so a lock-expired caller can't clobber a newer blob; cap restore size against
maxBytes before buffering.
- session-checkpoint: lchown + never follow symlinks when chowning a restored
(untrusted) checkpoint, so a symlinked entry can't re-own files outside the
workspace.
- lambda-client: throw when a MicroVM response omits microvmId instead of
returning '' (a partial RunMicrovm response would otherwise orphan a billable
VM behind getMicrovm('')/terminateMicrovm('')).
- router: skip runtime_session_hint validation in stateless mode (the field is
ignored there, so a malformed hint must not 400).
- v2/session: only run in the persistent workspace when THIS request carried a
valid X-Runtime-Session-Id, so a headerless request never inherits a prior
session's workspace/UID.
- entrypoint: only ever raise RLIMIT_NOFILE (guard so it never clamps a higher
host default down to 65536).
- secure-startup: warn (don't silently no-op) on affinity+http.
Deferred with reasons in the PR: zset-orphan (no consumer until the sweeper
PR), bindSessionWorkspace rebind race (prevented by the 1-VM-1-session
invariant), releaseLock swallow (TTL self-heals), /run applyRunHook (inert in
the hookless design), startupApiOnly policy placement (split-deploy design Q).
Collaborator
Author
|
Addressed the review findings in Fixed:
Deferred, with reasoning:
|
Collaborator
Author
|
@codex review |
- terraform: artifact + checkpoint buckets use SSE-S3 (AES256) so the s3:GetObject build role and the checkpoint access policy need no kms:Decrypt grant (SSE-KMS would AccessDenied on read). - terraform: build + runtime log policies grant both the log-group ARN and its stream form — stream-level actions (CreateLogStream/PutLogEvents) don't match the group ARN, which would fail builds with an empty stateReason. - terraform: validate artifact_bucket_name is non-empty when reusing an existing bucket (else the policy resolves to arn:aws:s3:::/*); bump required_version to >= 1.9 for cross-variable validation. - create-microvm-image.ts: cap the poll loop with a deadline (default 30m, MICROVM_BUILD_DEADLINE_MINUTES) and exit non-zero, so a wedged CREATING build can't hang a provisioning job forever. - README: add MINIO_PORT=443 to the S3 example (client defaults to 9000); scope the teardown sweep to VMs from this image's ARN so it can't terminate unrelated MicroVMs in a shared account.
Collaborator
Author
|
@codex review |
- terraform: include region in the artifact + checkpoint bucket names (S3 names are global, so a same-prefix second-region apply would collide). - docs: correct the checkpoint credential guidance. The MinIO-compatible client reads only static MINIO_ACCESS_KEY/SECRET and does not load task-role/IRSA creds, so attaching checkpoint_access_policy_arn to a task role alone does not work. Point operators at create_checkpoint_access_user (static keys) and note IRSA-aware loading as a follow-up. Fixed across variables.tf, outputs.tf, README.
Collaborator
Author
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8e2da55797
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| try { | ||
| const existing = await readRuntimeSessionRecord(runtimeSessionId); | ||
| const vm = await this.findOrLaunchSession(client, ctx, runtimeSessionId, existing, lockToken); | ||
| const result = await this.proxyExecute(client, vm, req, ctx, runtimeSessionId); |
Collaborator
Author
There was a problem hiding this comment.
Fixed in daa43b0. New executeOnSessionVm helper: on a reuse failure where the runner is unreachable (health/connection error — e.g. idlePolicy auto-terminated the suspended VM), it terminates the VM and removeRuntimeSession, so the next call relaunches + restores instead of reusing a dead VM still recorded RUNNING. A live-but-non-200 response leaves the warm VM and its record intact.
| } | ||
| this.sessionFiles.push(fileData); | ||
| this.generatedFiles.push({ id: newId, name: relativePath, path: fullPath }); | ||
| if (this.session) this.session.markSurfaced(relativePath, outputSignature); |
Collaborator
Author
There was a problem hiding this comment.
Fixed in daa43b0. The surfaced-mark is now recorded per-file (id → name+signature) during the scan and committed to session.markSurfaced only for files that upload successfully, in uploadGeneratedFiles. A dropped upload leaves the file eligible to surface next turn.
| cpuConfigurations: [{ architecture: 'ARM_64' as const }], | ||
| resources: [{ minimumMemoryInMiB: memory }], | ||
| additionalOsCapabilities: ['ALL' as const], | ||
| environmentVariables: { SANDBOX_USE_CGROUPV2: 'false' }, |
Collaborator
Author
There was a problem hiding this comment.
Fixed in daa43b0. create-microvm-image now takes --env-json / MICROVM_IMAGE_ENV_JSON, merged over SANDBOX_USE_CGROUPV2, so operators bake FILE_SERVER_URL / EGRESS_GATEWAY_URL / manifest env into the image at build time. README updated with the required keys and the "invalid /sessions/... URLs" rationale.
| const now = Date.now(); | ||
| const settled = await readRuntimeSessionRecord(runtimeSessionId); | ||
| const nextRecord = settled | ||
| ? await this.checkpointUnderLock(client, settled, runtimeSessionId, now, lockToken) |
Collaborator
Author
There was a problem hiding this comment.
Deferred — valid, but the fix is a design choice. checkpointUnderLock is already best-effort (never throws), so a checkpoint failure doesn't lose the result, but its latency still counts against JOB_TIMEOUT. Decoupling it (complete the job then checkpoint, or include the checkpoint budget in the public wait) needs a call on where checkpointing sits relative to job completion — raised with the maintainer. Not a correctness issue.
Comment on lines
+48
to
+51
| for await (const chunk of stream) { | ||
| chunks.push(chunk as Buffer); | ||
| } | ||
| return Buffer.concat(chunks); |
Collaborator
Author
There was a problem hiding this comment.
Fixed in daa43b0. CheckpointStore.get now takes maxBytes and stats the S3 object size before downloading (throws CheckpointTooLargeError if over), so we never Buffer.concat an oversized object. Removed the now-redundant post-buffer guard in restoreSession; added a MemoryCheckpointStore oversize test.
- Bind the session on checkpoint/restore (F1): the hookless path runs /session/restore before the first /execute, so the runner had nothing bound and every real restore 409'd, losing checkpoint state across VM expiry. The runner now binds from X-Runtime-Session-Id in the checkpoint/restore routes, and the backend sends that header on both proxied calls. - Clear/terminate a session VM on reuse failure or abort (F2/F5): on a dead reused VM (idlePolicy auto-terminated) or an aborted execute (runner keeps NsJail alive after the socket closes), terminate the VM and drop the record so the next call relaunches + restores instead of reusing a dead-or-dirty VM. A plain non-200 from a live runner leaves the warm VM intact. - Enforce checkpoint size before buffering (F7): CheckpointStore.get takes maxBytes and stats the S3 object first, so an oversized/stray checkpoint can't OOM the worker before the (now-removed, too-late) post-buffer guard. - Mark session outputs surfaced only after upload (F3): a dropped upload no longer permanently suppresses an unchanged file next turn. - Bake runner file/egress/manifest env into the image (F4): create-microvm-image takes --env-json so images can reach FILE_SERVER_URL / EGRESS_GATEWAY_URL instead of building invalid /sessions/... URLs. Deferred (P2, design choice): don't count checkpoint time in the client job timeout — decoupling checkpoint from the response path needs a call on where it sits relative to job completion; raised with the maintainer.
danny-avila
commented
Jul 7, 2026
| try { | ||
| const existing = await readRuntimeSessionRecord(runtimeSessionId); | ||
| const vm = await this.findOrLaunchSession(client, ctx, runtimeSessionId, existing, lockToken); | ||
| const result = await this.proxyExecute(client, vm, req, ctx, runtimeSessionId); |
Collaborator
Author
There was a problem hiding this comment.
Fixed in daa43b0. New executeOnSessionVm helper: on a reuse failure where the runner is unreachable (health/connection error — e.g. idlePolicy auto-terminated the suspended VM), it terminates the VM and removeRuntimeSession, so the next call relaunches + restores instead of reusing a dead VM still recorded RUNNING. A live-but-non-200 response leaves the warm VM and its record intact.
danny-avila
commented
Jul 7, 2026
| } | ||
| this.sessionFiles.push(fileData); | ||
| this.generatedFiles.push({ id: newId, name: relativePath, path: fullPath }); | ||
| if (this.session) this.session.markSurfaced(relativePath, outputSignature); |
Collaborator
Author
There was a problem hiding this comment.
Fixed in daa43b0. The surfaced-mark is now recorded per-file (id → name+signature) during the scan and committed to session.markSurfaced only for files that upload successfully, in uploadGeneratedFiles. A dropped upload leaves the file eligible to surface next turn.
danny-avila
commented
Jul 7, 2026
| cpuConfigurations: [{ architecture: 'ARM_64' as const }], | ||
| resources: [{ minimumMemoryInMiB: memory }], | ||
| additionalOsCapabilities: ['ALL' as const], | ||
| environmentVariables: { SANDBOX_USE_CGROUPV2: 'false' }, |
Collaborator
Author
There was a problem hiding this comment.
Fixed in daa43b0. create-microvm-image now takes --env-json / MICROVM_IMAGE_ENV_JSON, merged over SANDBOX_USE_CGROUPV2, so operators bake FILE_SERVER_URL / EGRESS_GATEWAY_URL / manifest env into the image at build time. README updated with the required keys and the "invalid /sessions/... URLs" rationale.
danny-avila
commented
Jul 7, 2026
| const now = Date.now(); | ||
| const settled = await readRuntimeSessionRecord(runtimeSessionId); | ||
| const nextRecord = settled | ||
| ? await this.checkpointUnderLock(client, settled, runtimeSessionId, now, lockToken) |
Collaborator
Author
There was a problem hiding this comment.
Deferred — valid, but the fix is a design choice. checkpointUnderLock is already best-effort (never throws), so a checkpoint failure doesn't lose the result, but its latency still counts against JOB_TIMEOUT. Decoupling it (complete the job then checkpoint, or include the checkpoint budget in the public wait) needs a call on where checkpointing sits relative to job completion — raised with the maintainer. Not a correctness issue.
danny-avila
commented
Jul 7, 2026
Comment on lines
+48
to
+51
| for await (const chunk of stream) { | ||
| chunks.push(chunk as Buffer); | ||
| } | ||
| return Buffer.concat(chunks); |
Collaborator
Author
There was a problem hiding this comment.
Fixed in daa43b0. CheckpointStore.get now takes maxBytes and stats the S3 object size before downloading (throws CheckpointTooLargeError if over), so we never Buffer.concat an oversized object. Removed the now-redundant post-buffer guard in restoreSession; added a MemoryCheckpointStore oversize test.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 16e6304c1f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+448
to
+449
| const token = await this.mintAuthToken(client, microvmId); | ||
| const deadline = Date.now() + this.config.launchTimeoutMs; |
There was a problem hiding this comment.
When LAMBDA_MICROVM_LAUNCH_TIMEOUT_MS is configured near or above the MicroVM auth-token lifetime for slow boots, a token minted once before readiness polling can expire while the loop is still waiting, so later health checks report a healthy-but-slow VM as MICROVM_UNHEALTHY. Refresh the token during long polls (and avoid carrying the execute token across the same wait) so slow launches don't fail solely because the proxy credential aged out.
Useful? React with 👍 / 👎.
Fresh MicroVMs lazy-page their rootfs, so the first touch of a large read-mostly asset is brutal: chdb's 408MB shared object takes 30-120s to import cold (it blew the 120s exec timeout in the field, presenting as a silent hang) but ~250ms warm. Two mitigations: - SANDBOX_WARMUP_COMMAND: optional command spawned once at sandbox-API startup, detached and outside any job sandbox, so it pre-faults heavy assets into the page cache during the boot window instead of the user's first tool call, without ever holding a session lock or blocking readiness. Best-effort by design (failures logged, ignored). Set e.g. SANDBOX_WARMUP_COMMAND="python3 -c 'import chdb'" via the image env. - LAMBDA_MICROVM_IDLE_SECONDS default 300 -> 1800: keep a session's VM fully RUNNING (RAM + page cache intact, ~0.3s follow-ups) across a realistic conversation gap before suspending; still env-configurable.
Boot-time deaths (VM enters TERMINATED/TERMINATING before becoming
ready) are fast provider-side transients — observed several times a day
in the field — and the backend previously surfaced each one straight to
the caller as MICROVM_LAUNCH_FAILED (HTTP 503) on the user's tool call.
Mark that specific failure transient on SandboxBackendError and have
launch() retry exactly once with a fresh clientToken (RunMicrovm is
idempotent per token, so reusing the original would return the same
dead VM). Throttles, aborts, and poll-deadline timeouts stay
non-retried: a throttle retry adds pressure to a poisoned bucket and a
timeout retry doubles a 60s wait against the job budget. Retries are
visible via the microvmLaunches{outcome=retried} counter.
The table claimed both streams truncate at the cap. stdout overflow actually SIGKILLs the job (status OL) — stdout is the result, so a runaway producer is cut off — while stderr truncates and the job continues. Also note that every shipped compose/helm config sets 65536, so the 1024 fallback only applies to a bare runner.
MicroVMs have internet-only egress, so the runner's pull-based input priming has nothing reachable to pull from — uploads authorized into the internal file server never reached the sandbox. Deliver them push-model instead, over the same authed proxy channel as a checkpoint restore: - runner: additive POST /api/v2/session/files extracts a tar.gz overlay into the bound session workspace (restore's traversal hardening, but never a wipe — failure leaves existing session state untouched). A reserved manifest member registers delivered files as primed, so the execute's own priming reuses the on-disk copy (same-id reuse) and later turns suppress unchanged inputs from the output scan. - service: buildSessionFilesArchive fetches the request's by-ref files from the internal file server, stages them under session/ with the primed-files manifest, and pushFiles delivers the archive before the execute under the held session lock. Build failures throw before any bytes reach the VM (warm VM survives); a failed push after transfer began recycles the VM like a failed push-restore. Known limitations (follow-ups): delivery re-pushes on every exec (no service-side dedupe against the runner's primed state yet), and the stateless path still has no delivery leg.
Push-model deployments fetch input bytes control-plane-side, so the file server only needs to be reachable from the co-located service — binding 127.0.0.1 keeps the object routes off the network entirely. Unset preserves the historical all-interfaces bind.
Addresses the P1 review findings on the push-model delivery and session machinery: - delivery dedupe: writable refs already delivered to the live workspace are recorded on the session (delivered_files + delivered_at) and never re-pushed — a later turn re-sending the same ref can no longer overwrite in-place modifications the sandbox made. After a relaunch the list is trusted only when the restored checkpoint post-dates the last delivery. Read-only refs (X-Read-Only from the file server) are the deliberate exception: re-delivered every exec and primed read-only by the runner, mirroring the pull model's re-download rule. - checkpoint restore fails closed: a transient fetch failure now recycles with a retryable 503 instead of executing on an empty workspace whose post-run checkpoint would prune the last good snapshot (transient S3 blip -> permanent data loss). - fence semantics: a LOST lock renewal (vs a transient renew error) aborts the in-flight critical path via AbortSignal instead of merely stopping the heartbeat; fenced failures never terminate the VM the new lock holder may be using. - aborted or failed pushes always recycle the VM (partial extraction must not be reused); archive construction gains a file-count cap, a cumulative uncompressed-byte budget, and abort-signal support; fetch errors log sanitized details instead of raw axios errors that carried the internal service token. - hintless requests never land on a session: affinity degrades them to stateless one-shots, strict rejects them (router 400, worker hard fail) instead of silently sharing the per-user default workspace. - runner: rebinding a different runtime session id is refused outright (fail-closed; no async-wipe race over the shared directory); a failed workspace wipe retains the pinned UID instead of releasing it over a quarantined directory; tar spawn errors reject instead of crashing the process; sha256File opens via fsp.open for typed O_NOFOLLOW.
Second review round on the push-model delivery. Several of these were introduced by the previous hardening commit: - rebind rejection is now enforced: bindSessionFromHeader propagates the refusal (409) and /execute conflicts instead of silently running under the previously bound session's workspace and UID. - read-only pushed inputs are usable again: the runner marks control-plane deliveries as fresh-for-this-exec, so priming reuses the pristine copy rather than falling through to the pull path a push-model backend cannot reach. Delivered read-only files also get the pull path's root-owned 0444 protection, and reuse carries the read-only bit so modifications are dropped by contract. - delivery keys include the destination path: the same object under a new filename is a file the workspace does not have yet and is delivered. - delivery metadata survives relaunch: the record is reconciled even when every ref is skipped, so a restored session no longer re-pushes originals over restored in-place edits. - fencing actually stops concurrent mutation: a fenced worker terminates the VM (closing the socket does NOT stop the runner's NsJail child), and persistent renew errors fence once the lease can no longer be proven held, not just on an explicit lost token. - affinity lock contention no longer silently drops by-reference inputs: the stateless fallback runs only for requests without them; otherwise the call fails retryably as BUSY. - SandboxBackendError sanitizes its cause at construction, so no logging path can serialize axios config carrying the internal service token, MicroVM auth headers, or the pushed archive body. - manifest failures fail the delivery (500) instead of acknowledging a push the next execute cannot use; unparseable content under the reserved name is left alone as user data, and a ref claiming the reserved name is refused at build time. - restore metadata replaces rather than merges, so stale primed/surfaced entries cannot suppress real files after a repeated restore. - tar 'close' rejections are observed immediately: a spawn failure used to crash the runner with an unhandled rejection after answering 500. - typed O_NOFOLLOW reads via fsp.open (clears the remaining src/ tsc errors in the runner).
…plane state Live testing found the overwrite protection was still reachable: a delivery failure recycles the VM and drops the session record, taking delivered_files with it, so the retry re-pushed originals over the checkpoint-restored in-place edits. Redis dedupe cannot be the correctness mechanism — it dies with the record (recycle, failover, flush) while the edits live on in the checkpoint. Enforce the invariant where the data is instead: deliveries now extract into a 0700 staging dir beside the workspace and merge file-by-file. A WRITABLE input whose on-disk content differs from its primed baseline is kept as-is (the sandbox's edit wins over a re-pushed original); read-only inputs are always restored to pristine bytes, as the pull path does. The control-plane dedupe stays as a bandwidth optimization. Staging also tightens the delivery: only regular files found inside staging are ever moved (symlinks skipped, containment re-checked), and every manifest entry must correspond to a delivered file, so an incomplete archive fails the delivery rather than stranding the execute on the unreachable pull path.
Round 3, runner half. The staging merge added in 82bb551 performed privileged mkdir/rm/rename against paths validated only lexically, and committed files before finishing validation. - Reuse the pull path's proven no-follow ancestor walk instead of hand-rolling one: Job.ensureDirNoFollow is extracted to workspace-isolation as a shared helper (identical semantics, plus an optional identity so directories it creates are owned by the session rather than left root-owned and unwritable by the sandbox). A prior turn's sandbox code can plant a symlink in the persistent workspace, and mkdir -p/rename would follow it and write outside as root. - Validate everything before touching live state: canonical names via the same validateFilePath the pull path uses, plus a unique one-to-one correspondence between manifest entries and staged regular files. A missing, duplicate, extra, or escaping member now fails the delivery with the workspace untouched — including a manifest-less archive. - Preserve a sandbox edit only when the incoming ref is the SAME file (id + storage session) the edit was made to; a different ref at that path must land and re-prime, or the execute runs against bytes whose identity matches nothing it declared. - Replace directories recursively (plain rm could not, wedging that ref) and let rename replace files/symlinks atomically without a pre-unlink. - A failure part-way through the commit phase cannot be rolled back, so quarantine the workspace: the runner refuses further execution until the control plane recycles the VM and restores the last checkpoint.
Delivery was a SECOND writer into the session workspace, duplicating what the pull path already does — name validation, no-follow ancestors, ownership, read-only protection, priming, modification detection. Every delivery defect across three review rounds was a bug in one of those copies. This removes the duplication instead of patching it again. The push now fills a runner-local input cache keyed by a digest of (storage session, object id), outside any workspace. Priming resolves a ref from that cache and otherwise fetches over HTTP; a cache hit is presented as the Response a fetch would have returned, so every downstream step is byte-identical for pushed and pulled inputs. The workspace keeps exactly one writer. Consequences, all previously open findings: - No caller-controlled path ever reaches the filesystem during delivery, so the symlinked-ancestor class cannot recur here at all. - Re-delivery cannot revert a sandbox edit, because delivery no longer touches the workspace; reusePrimedInput decides, as it always has. - Dedupe is a probe answered by the VM, not Redis bookkeeping, so it stays correct across recycles, failover and flushes — and the checkpoint-coverage timestamp comparison is gone entirely. - Stateless one-shots receive by-reference inputs too (the cache is keyed by object, not session); they previously ran with inputs missing. - Session lock contention is now always a retryable BUSY: a session-bound request depends on workspace state a cold VM lacks, whether or not this particular payload carries refs. - Read-only, duplicate-name and manifest-shape handling all collapse into the single existing implementation. Net effect on the runner: mergeDeliveredFiles, the manifest protocol, the staging merge, quarantine plumbing and the delivered_files registry fields are deleted.
There is no global express.json (see index.ts); parsers are per-route, so the probe endpoint saw an unparsed body and rejected every ref list with 'refs must be an array'. Only live execution surfaced this — route wiring is invisible to the unit suites.
The cache was a dot-directory inside SANDBOX_WORKSPACE_ROOT, where the stale-workspace reaper treats every entry as a workspace — it deleted pending inputs between the push and the execute they were pushed for, so a live exec reported 'No such file' for files the control plane had successfully delivered. Unit and route tests both passed; only the real sandbox showed it, and reapStaleWorkspaces reproduced it on demand. Moved to a sibling directory (SANDBOX_INPUT_CACHE_DIR, default <root>-inputs), which also keeps it off every nsjail mount, and pinned the invariant with a regression test that runs the reaper over a seeded cache.
The two ends computed the SAME contract differently — the runner joined (storage session, id) with a literal NUL byte, the control plane with a space — so every pushed object landed under a name no lookup could ever produce. Both unit suites passed because each was self-consistent; the route test and the priming test used one side only. A live execution reported 'No such file' for files that had been delivered successfully. Both now use an explicit \u0000 escape (no raw NUL in source, which also made grep treat the file as binary), and both suites assert the same golden vector so the contract cannot silently drift again.
| const endpointBase = normalizeMicrovmEndpoint(vm.endpoint ?? ''); | ||
| const mintToken = () => this.mintAuthToken(client, vm.microvmId); | ||
| try { | ||
| const missing = await probeInputs( |
There was a problem hiding this comment.
🟠 High sandbox-backend/lambda-microvm.ts:636
deliverInputs probes a freshly-launched VM's input cache before the runner's HTTP listener is ready. A VM in RUNNING state can still be booting its app, and the readiness wait only happens later in proxyExecute — so probeInputs hits a VM whose listener isn't up yet, causing normal boot-race failures (connection refused / timeout) for stateless executions and checkpoint-disabled sessions that carry by-reference files, instead of polling until the runner is ready. Consider waiting for runner readiness before deliverInputs, the way findOrLaunchSession does before restore.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @service/src/sandbox-backend/lambda-microvm.ts around line 636:
`deliverInputs` probes a freshly-launched VM's input cache before the runner's HTTP listener is ready. A VM in `RUNNING` state can still be booting its app, and the readiness wait only happens later in `proxyExecute` — so `probeInputs` hits a VM whose listener isn't up yet, causing normal boot-race failures (connection refused / timeout) for stateless executions and checkpoint-disabled sessions that carry by-reference files, instead of polling until the runner is ready. Consider waiting for runner readiness before `deliverInputs`, the way `findOrLaunchSession` does before restore.
The cache is keyed by OBJECT, but the pushed metadata carried the file server's filename and the runner emitted it as Content-Disposition. Every ref for that object therefore resolved to one name, so requesting the same object at a second path wrote it to the FIRST path instead — overwriting a file the sandbox had edited in an earlier turn. Live testing caught it; the unit suites had encoded the wrong behaviour as expected. Metadata now carries only object-level facts (read-only), and priming falls back to each ref's requested name. Covered by a prime test that serves one cached object to two destinations.
Collaborator
Author
|
@codex review |
|
Macroscope has since reviewed this pull request. An earlier review was skipped by a cost limit; a review has now completed, so that notice no longer applies. |
Collaborator
Author
|
@codex review |
Collaborator
Author
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d8d9f110bf
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+145
to
+149
| runtimeSessionId = resolveRuntimeSessionIdForRequest({ | ||
| mode: env.RUNTIME_SESSION_MODE, | ||
| storageNamespace: identity.storageNamespace, | ||
| canonicalUserId: identity.canonicalUserId, | ||
| hint, |
There was a problem hiding this comment.
When CODEAPI_RUNTIME_SESSION_MODE=strict, synthetic health/smoke probes are deliberately sessionless (the worker exemption for isSyntheticJob never gets reached), but the router still calls resolveRuntimeSessionIdForRequest with no hint and returns a 400 before enqueueing. In strict-mode deployments, any synthetic /exec probe that omits runtime_session_hint will fail even though the worker path is designed to allow it; skip runtime-session resolution for isSyntheticRequest or otherwise exempt it here.
Useful? React with 👍 / 👎.
| "lambda:CreateMicrovmAuthToken", | ||
| "lambda:TerminateMicrovm", | ||
| ] | ||
| resources = [local.microvm_image_arn] |
There was a problem hiding this comment.
With the Terraform policy as written, the worker role grants RunMicrovm/GetMicrovm/CreateMicrovmAuthToken/TerminateMicrovm only on the image ARN, but these control-plane calls authorize against MicroVM instance ARNs (arn:...:microvm:<id>; AWS's least-privilege example uses microvm:* for these actions). Deployments that attach this policy will hit AccessDenied as soon as the Lambda backend tries to launch or manage a VM, so scope these actions to the MicroVM ARN pattern instead of local.microvm_image_arn.
Useful? React with 👍 / 👎.
Comment on lines
+827
to
+829
| } catch (error) { | ||
| throw new SandboxBackendError('MICROVM_UNHEALTHY', 'Session input source fetch failed', error); | ||
| } |
There was a problem hiding this comment.
When a by-reference input cannot be fetched or packed after the probe, such as an oversized delivery batch or a stale file ref returning 404/403 from the file server, buildInputBatch raises SessionFilesError but this wrapper converts it to MICROVM_UNHEALTHY. The worker then exposes a retryable 503 microvm_unhealthy even though the failure is in the request/input source, so callers get a sandbox-outage signal instead of a normal input error; preserve or map SessionFilesError separately rather than wrapping it as VM health here.
Useful? React with 👍 / 👎.
Comment on lines
+93
to
+94
| if (env.RUNTIME_SESSION_MODE === 'strict' && !isSyntheticJob && !job.data.runtimeSessionId) { | ||
| throw new Error('strict runtime session mode requires a runtimeSessionId on the job'); |
There was a problem hiding this comment.
When CODEAPI_RUNTIME_SESSION_MODE=strict is enabled, this guard rejects every non-synthetic job that lacks runtimeSessionId, but the programmatic/replay enqueue paths still never populate that field. In a strict Lambda MicroVM deployment, PTC/replay executions therefore fail before reaching the sandbox even though /exec jobs get a session id; derive/pass a runtime session for those producers or exempt unsupported programmatic jobs explicitly.
Useful? React with 👍 / 👎.
| const raw = process.env.MINIO_ENDPOINT ?? 'localhost'; | ||
| const protocol = process.env.MINIO_USE_SSL === 'true' ? 'https:' : 'http:'; | ||
| const parsed = new URL(raw.includes('://') ? raw : `${protocol}//${raw}`); | ||
| if (!parsed.port) parsed.port = process.env.MINIO_PORT || '9000'; |
There was a problem hiding this comment.
When checkpoints are configured against AWS S3 with a scheme-qualified endpoint such as https://s3.us-east-1.amazonaws.com, this helper still appends :9000 whenever MINIO_PORT is unset. Startup validation only requires MINIO_ENDPOINT and a bucket, so the worker can boot with what looks like a valid S3 URL and then every checkpoint/restore request goes to the wrong port; leave scheme-qualified endpoints on their URL default port or only apply the MinIO 9000 default to bare/local endpoints.
Useful? React with 👍 / 👎.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds AWS Lambda MicroVMs as an optional, config-gated stateful execution backend for the Code Interpreter, giving perceived-indefinite statefulness (a warm per-session workspace plus checkpoint/restore across the VM's 8h lifetime) without changing the legacy semi-stateless HTTP path.
Design
SandboxBackendseam is extracted from the worker dispatch. Thehttpbackend is byte-identical to today.CODEAPI_SANDBOX_BACKEND=lambda-microvmopts in, and@aws-sdk/*is lazily imported so http deployments never load it.runtime_session_id = hash(storageNamespace, canonicalUserId, hint)plus a Redis registry (SET NX locks, Lua CAS release, generation fencing) pins one VM per session.CODEAPI_RUNTIME_SESSION_MODE=stateless|affinity|strict./mnt/dataand pinned UID across calls (prime dedup and output diffing), gated bySANDBOX_SESSION_WORKSPACE_ENABLED(Lambda image only) plus a per-request signal.idlePolicy(autoResume) handles idle suspend/resume, so the sweeper shrinks to registry hygiene.Hookless session binding
Session mode is delivered per request via an
X-Runtime-Session-Idheader rather than a/runlifecycle hook. Lambda's image build hooks only route on the snapshot-compatible base container image, and enabling any runtime hook forces the/readybuild hook, which never reaches a stock container's listener (builds then fail at the ready timeout). Hookless image builds are reliable; checkpoint/restore already runs over plain HTTP endpoints, and idle suspend/resume is native.Proven live
Real MicroVM: launch ~3s, suspend/resume ~2.2s with process continuity, auto-resume ~1.2s. Two-turn E2E on a real hookless MicroVM: with the header,
/mnt/datapersists across separate/executecalls (42 read back); without it, fresh-per-job (ABSENT).Tests
~630 api and service tests (ioredis-mock, fake AWS client, and Bun.serve fakes).