Add OCI image support: pull, unpack, run, prune, status, policy

[Max042004]

This PR lands the full elfuse OCI image support. It supersedes the
original Phase 1 scope of this PR (CLI scaffold + pull/inspect) and
now covers Phases 1-4 plus the post-Phase-3 improvements plan: image
layout alignment, GC/prune, layer + stack snapshot caches, store
status, parallel pull, registry policy.json, and a heavy-mode compat
matrix.

Scope

• Pull / inspect — content-addressable blob store, HTTPS + bearer
  token, OCI index walk to the linux/arm64 leaf manifest, partial-
  store-aware inspect renderer.
• Unpack — tar reader (ustar + PAX x/g records), gzip + decode-
  only vendored zstd, whiteout-aware layer apply (typeflag '1'/'2'/'5'
  • .wh.* markers), per-image sysroot on a case-sensitive APFS
    sparsebundle.
• Run — elfuse oci run clones the unpacked tree via clonefile(2),
  honors Entrypoint / Cmd / Env / WorkingDir / User, and reuses the
  existing elfuse launch path so a dynamically-linked guest binary
  runs through the same shim + syscall surface as the non-OCI mode.
• Lifecycle — oci prune with --older-than / --keep-bytes;
  layer + stack prune sweep; oci status (text + --json);
  oci rebuild-cache for pre-snapshot stores.
• Performance — parallel blob fetch with HTTP Range resume;
  per-layer raw snapshot cache; ChainID stack snapshot cache; APFS
  COW clone-rootfs reuse between runs.
• Policy — podman / skopeo-style policy.json + registries.d
  overlay (per-registry insecure / ca_bundle / auth_file). CLI flags
  override; loopback-only --insecure.
• Test coverage — 25 OCI unit suites (test-oci-*), compat-shell
  smoke (tests/test-oci-compat.sh), and an opt-in heavy mode
  (OCI_COMPAT_TEST=1) that drives three layered fixtures
  (alpine-shaped, busybox-shaped hardlink dispatch, two-layer
  whiteout) end-to-end through a freshly-provisioned scratch
  sparsebundle.

Manual smoke test (docker.io/library/python:3.12)

A real end-to-end pull-and-run against a mainstream multi-layer glibc
image. The image's default Entrypoint is docker-entrypoint.sh (a
shell script, which elfuse does not execute), so the commands below
override --entrypoint to the python3 binary directly.

make elfuse
SCRATCH=$(mktemp -d)
echo "store: $SCRATCH"

# 1. Pull (~400 MB across 7 layers, ~3 minutes on a fast link).
#    If your terminal mishandles CSI cursor-up and the progress
#    output stacks duplicate rows, prepend ELFUSE_OCI_PROGRESS=plain
#    to fall back to one summary line per blob.
./build/elfuse oci pull --store "$SCRATCH" python:3.12

# 2. Offline inspect: image index -> linux/arm64 manifest -> config
#    runtime block (Entrypoint / Cmd / Env / WorkingDir / User).
./build/elfuse oci inspect --store "$SCRATCH" python:3.12

# 3. Cold run. First invocation triggers layer unpack onto the
#    sysroot APFS sparsebundle, then clone-rootfs, then launch. The
#    unpack step dominates the ~50 s wall on a fresh store.
./build/elfuse oci run --store "$SCRATCH" \
    --entrypoint /usr/local/bin/python3 python:3.12 \
    -c 'print("hello from elfuse", 1+2)'
# expected stdout:  hello from elfuse 3

# 4. Warm run. clone-rootfs reuses the unpacked image tree, so wall
#    drops to ~2 s and is dominated by VM bring-up + dynamic-linker
#    bring-up + Python interp init.
./build/elfuse oci run --store "$SCRATCH" \
    --entrypoint /usr/local/bin/python3 python:3.12 \
    -c 'import sys, platform; print(sys.version); print(platform.platform()); print(platform.machine())'
# expected stdout:  Python 3.12.x ... / Linux-<kernel>-aarch64-with-glibc2.41 / aarch64

# 5. stdlib smoke. Confirms json + math + f-string formatting all
#    flow through the emulated syscall surface.
./build/elfuse oci run --store "$SCRATCH" \
    --entrypoint /usr/local/bin/python3 python:3.12 \
    -c 'import json, math; print(json.dumps({"pi": round(math.pi, 5), "ok": True}))'
# expected stdout:  {"pi": 3.14159, "ok": true}

Performance characterization (vs OrbStack)

Measured on Apple M4 / macOS 15.4.1 (Darwin 24.4.0). OrbStack 2.1.3
acts as the ground-truth aarch64-linux runtime: it executes the same
docker.io/library/python:3.12 image inside a Virtualization.framework-
backed Linux VM with a real Linux kernel, so the comparison isolates
the cost of elfuse's user-mode ABI emulation against a native syscall
surface.

Pure CPU (factorial big-int multiply, no syscall)

import sys, math, time
sys.set_int_max_str_digits(0)   # Python 3.12 default cap is 4300 digits
N = 200000
t = time.perf_counter()
f = math.factorial(N)
s = sum(int(d) for d in str(f))
print("fact(%d) digit_sum=%d digits=%d compute=%.3fs" %
      (N, s, len(str(f)), time.perf_counter() - t))

Each engine ran twice; the second is warm. compute is the
time.perf_counter() delta inside Python (pure interpreter +
big-int multiply work); real is the outer wall (includes engine
startup); startup ≈ real - compute.

Engine	run	compute (s)	real (s)	startup (s)
elfuse	1	0.791	3.72	2.93
elfuse	2 warm	0.804	3.35	2.55
orbstack	1	0.792	1.10	0.31
orbstack	2 warm	0.796	0.97	0.17

Both engines emit digit_sum=4154076 digits=973351 — correctness
parity confirmed. Pure compute ratio: 1.01× (within measurement
noise). HVF runs guest aarch64 instructions directly so big-int
multiply + Python bytecode dispatch pay zero translation overhead.
Startup ratio: 15.0× (constant ~2.5 s for elfuse vs ~0.17 s for
orbstack), independent of N — verified separately at N=50000 where
both compute drops to ~0.14 s but elfuse startup stays at 2.53 s.

Syscall density (Python loop hammering syscalls)

import os, time
N_BASE = 1_000_000
N_READ = 100_000

def time_loop(label, fn, n):
    fn(min(n // 100, 10_000))   # warm-up
    t = time.perf_counter()
    fn(n)
    return label, time.perf_counter() - t, n

def baseline(n):
    for _ in range(n): pass

def getppid(n):
    g = os.getppid
    for _ in range(n): g()

def clock_ns(n):
    g = time.monotonic_ns
    for _ in range(n): g()

def urandom_read(n):
    fd = os.open("/dev/urandom", os.O_RDONLY)
    try:
        rd = os.read
        for _ in range(n): rd(fd, 1)
    finally:
        os.close(fd)

results = [
    time_loop("baseline (pass)",              baseline,     N_BASE),
    time_loop("getppid",                      getppid,      N_BASE),
    time_loop("clock_gettime (monotonic_ns)", clock_ns,     N_BASE),
    time_loop("/dev/urandom 1B read",         urandom_read, N_READ),
]
base_per = results[0][1] / results[0][2]
for label, secs, n in results:
    per = secs / n
    overhead = (per - base_per) * 1e6 if label != "baseline (pass)" else 0.0
    print("%-38s total=%.3fs n=%d per=%.3fus  syscall_overhead=%.3fus" %
          (label, secs, n, per * 1e6, overhead))

syscall_overhead strips the Python loop interpreter cost (measured
from the baseline band) so the residual is the pure trap+return
cost of a single syscall.

Band	elfuse (μs/call)	orbstack (μs/call)	ratio
baseline (pass)	0.007	0.007	1.0×
getppid	0.960	0.091	10.5×
clock_gettime (monotonic_ns)	1.006	0.018	55.9×
/dev/urandom 1B read	1.704	0.210	8.1×

getppid is the cleanest measurement: no kernel work, just trap +
return. elfuse pays roughly 1 μs per syscall versus ~0.1 μs native.
Rough HVF round-trip breakdown: vCPU state sync ~200 ns, Linux→macOS
semantics ~100 ns, the macOS syscall itself ~100 ns, errno + sync
back ~100 ns, HVF re-entry + ERET ~500 ns. This 1 μs floor is the
structural ceiling for any elfuse syscall path.

vDSO observation — time.monotonic_ns should hit the synthetic
vDSO under src/core/vdso.{c,h} and skip the trap (orbstack does, at
0.018 μs), but the measured 1.006 μs matches the trapping baseline.
elfuse's vDSO entry is not being picked up by glibc 2.41 in this
image. This is an existing optimization opportunity unrelated to the
scope of this PR; left untouched here so the patch series stays
focused on image-distribution and runtime correctness.

Wall-clock model

For a pure-CPU workload of compute time W:

elfuse_total   ≈ 2.5 s + W
orbstack_total ≈ 0.17 s + W

W	elfuse	orbstack	ratio	scenario
0.1 s	2.6 s	0.27 s	9.6×	CLI one-shot
1 s	3.5 s	1.17 s	3.0×	short script
10 s	12.5 s	10.17 s	1.23×	medium task
60 s	62.5 s	60.17 s	1.04×	batch job

elfuse is competitive for long-running workloads (where the constant
startup amortizes out) and a known tradeoff for short CLI one-shots
where startup dominates total wall.

Known limitations

• fork() followed by execve() of a dynamically-linked ELF crashes
  in the child during dynamic-linker bring-up. This blocks Python's
  subprocess.run([...other_dynamic_binary...]), shell pipelines that
  spawn external binaries, and timeout(1). Single-process Python
  workloads, stdlib computation, and file I/O are unaffected.
• Multi-arch image selection is hardcoded to linux/arm64. There is
  no --platform flag; cross-arch image support is out of scope for
  this PR.
• pull progress uses CSI cursor-up + clear-line for in-place
  redraw. Terminal panes that ignore those escapes show stacking
  rows; set ELFUSE_OCI_PROGRESS=plain to disable the redraw and
  emit one summary line per blob instead.

---

Summary by cubic

Adds full OCI image lifecycle to elfuse: pull, inspect, unpack, clone, run, prune, rebuild-cache, and status, with parallel/resumable downloads, a content‑addressable store + caches, and a runtime path to execute images. Vendors cJSON and decode‑only zstd to keep builds self-contained, and extracts a shared VM launcher for oci run.

• New Features

  • CLI: oci pull|inspect|unpack|clone|run|prune|rebuild-cache|status; pull adds progress and --refresh; status --json.
  • Registry/store: HTTPS via libcurl with bearer/basic auth, custom CA, loopback‑only --insecure; content‑addressable blob store; oci-layout marker; pins in OCI index.json.
  • Unpack/caches: ustar/PAX tar with gzip/zstd; whiteouts; case‑sensitive APFS sysroot; per‑run rootfs via clonefile(2); raw layer and ChainID stack caches; parallel fetch + HTTP Range resume.
  • Runtime: PATH resolver; image‑config User name/group lookup; inject /etc/{resolv.conf,hosts,hostname}; emulate /dev/{full,console} and basic /proc; multi‑arch index resolves to linux/arm64; ELFUSE_OCI_PROGRESS=plain fallback.
  • Policy/inspect: podman/skopeo‑style policy.json with registries.d overlay; CLI flags override; inspect shows runtime fields and cross‑image dedup stats.

• Migration

  • Pins moved to OCI index.json; store auto‑migrates from refs/ on open.
  • layers/ cache schema v2; first open wipes legacy v1 cache entries (blobs/images untouched).
  • Vendors decode‑only zstd and cJSON; uses system zlib and libcurl.

^{Written for commit 426d7f6. Summary will update on new commits. Review in cubic}

[jserv]

Rebase onto the latest main branch and squash/rework the commits into fewer, cleaner ones.