vm: cheaper call convention — one-allocation register files, static-arity opcodes, call_self fusion

[davydog187]

Last of the perf stack. The branches it stacked on — #400, #401, #403 —
have all landed, so this is now rebased directly onto main and carries
only its own seven commits. The base column below is what is now main
(the old perf/codegen-peephole tip), so the ratios still read as
this PR against main.

Three changes to the call convention, in ascending risk, one commit each.

1. One-allocation callee register files

Every compiled-closure call built the callee's register tuple twice over:
Tuple.duplicate/2 for the blank file, then one setelement per argument,
each copying the whole tuple again. Generated mkregs/4 clauses build the
finished tuple in a single literal construction for the small (width,
parameter-count) shapes ordinary Lua functions have. A parameterless callee
now allocates nothing at all — an all-nil tuple is a compile-time literal.

Wider shapes keep the duplicate-and-copy path, and the grow_regs/2 growth
contract for vararg and multi-return writes is untouched.

2. Static-arity call opcodes

arg_count is fixed at encode time, so the 0-, 1-, and 2-argument forms of
each fixed-result call get their own tag (70–75). Their handlers read
the arguments at constant offsets into the caller's registers and build the
callee's file as one literal tuple: no copy loop, no min/2 clamp against
the callee's parameter count, no blank tuple to overwrite. Wider arities
keep the generic opcodes.

name_hint and the baked source line ride along unchanged, so tracebacks
and native-call error attribution are identical. The interpreter is
untouched here: this is an encoding choice inside the dispatcher's own
representation, and both engines still run the same instruction stream.

The non-compiled-closure branches of the call handlers moved into shared
call_one_bridge/17 / call_zero_bridge/16 helpers, so the generic and
static-arity opcodes keep one copy of the interpreter and native bridges
between them.

3. call_self fusion

A local function reaches its own name through an upvalue cell, so every
self-call loads the closure out of the cell into a scratch register before
calling it. When the cell provably can only ever hold the closure that is
already running, that load is pure overhead and the call needs no callee at
all: both engines already hold the prototype and its upvalue tuple.

Lua.Compiler.Peephole proves it (rule 7), and every step fails closed:

• the parent binds the child with the shape codegen emits for
  local function, and writes that register nowhere else in the function;
• the parent only ever reads it to call it, and the scratch register the
  closure passed through dies before anything reads it back;
• inside the child the self-upvalue is only ever loaded to be called, is
  never assigned, and is captured by no nested prototype — so neither the
  value nor the cell behind it can reach code that could rebind it,
  debug.setupvalue included;
• the child is not vararg, and the function contains no goto.

Refused, and tested as refused: mutual recursion, a reassigned
local function, local f; f = function() … f() … end, a name a second
closure captures (with or without reassigning it), a name passed as a value
or handed to pcall, a vararg body, a body using goto.

The dispatcher's handler re-enters proto.bytecode with the upvalues it is
already carrying. It is a full call in every other respect: it pushes a
frame, ticks the instruction budget, checks the call depth at the same point
with the same depth, and starts the callee with an empty open-upvalue map —
so tracebacks, debug.getinfo, and the max_call_depth trip point are
identical to a generic call's. The interpreter reconstructs the closure the
cell holds and runs an ordinary call; one test pins that path (a
self-recursive body containing and/or still falls back off the
dispatcher, so :call_self runs on the interpreter there).

Emitted bytecode for fib

local function fib(n)
  if n < 2 then return n end
  return fib(n-1) + fib(n-2)
end

before (main) — 10 opcodes:

{65, 2, 0, 2}                       less_than_k  r2 = n < 2
{24, 2, {{27, 0}}, {}}              test
{6, 1, 1}                           get_upvalue  r1 = fib
{63, 2, 0, 1, {:local, "n"}}        subtract_k   r2 = n - 1
{26, 1, 1, {:upvalue, "fib"}, 3}    call_one     base 1, 1 arg
{6, 2, 1}                           get_upvalue  r2 = fib
{63, 3, 0, 2, {:local, "n"}}        subtract_k   r3 = n - 2
{26, 2, 1, {:upvalue, "fib"}, 3}    call_one     base 2, 1 arg
{9, 3, 1, 2, nil, nil}              add
{27, 3}                             return_one

after — 8 opcodes, and neither call touches the upvalue table:

{65, 2, 0, 2}                       less_than_k
{24, 2, {{27, 0}}, {}}              test
{63, 2, 0, 1, {:local, "n"}}        subtract_k
{76, 1, 1, 1, {:upvalue, "fib"}, 3} call_self    base 1, 1 arg, 1 result
{63, 3, 0, 2, {:local, "n"}}        subtract_k
{76, 2, 1, 1, {:upvalue, "fib"}, 3} call_self
{9, 3, 1, 2, nil, nil}              add
{27, 3}                             return_one

A non-self statement call shows change 2 on its own:
print(x) goes from {25, base, 1, hint, line} (call_zero, arg count in
slot 3) to {74, base, hint, line} (call_zero_1).

Measured

MIX_ENV=prod, one tree per commit built from the stack, runs interleaved
across all four builds so machine drift hits each equally; min of 3 passes ×
7 rounds each. Ratios are speedups (higher is faster); total is this PR
against base, which is now main.

workload         base        +c1        +c2        +c3        c1      c2      c3   total
calls3        3068.1us    2847.2us    2832.2us    2838.4us    1.078x  1.005x  0.998x  1.081x
closures      2337.1us    2334.9us    2369.1us    2330.2us    1.001x  0.986x  1.017x  1.003x
fib15          281.5us     247.4us     236.7us     207.3us    1.138x  1.045x  1.142x  1.358x
fib20         3102.6us    2767.0us    2583.2us    2326.6us    1.121x  1.071x  1.110x  1.334x
loopself        26.1us      24.7us      23.3us      19.5us    1.058x  1.060x  1.198x  1.343x
mutual        3146.8us    2871.2us    2851.7us    2827.9us    1.096x  1.007x  1.008x  1.113x
oop           1546.0us    1534.8us    1521.8us    1515.0us    1.007x  1.008x  1.005x  1.020x
tables         558.1us     562.8us     565.1us     549.5us    0.992x  0.996x  1.028x  1.016x

• fib15 1.36×, fib20 1.33×, loopself (tail-recursive
  accumulator) 1.34×, mutual (mutual recursion, which never fuses)
  1.11×, calls3 (3-deep non-recursive call chain) 1.08×.
• closures, oop, and tables are call-light or table-bound and land at
  1.00–1.02× — no regression, no win.
• A second independent interleaved run agreed on every workload within ~3%
  (fib15 1.32×, fib20 1.26×, loopself 1.30×, calls3 1.10×, mutual
  1.11×, the rest flat). This is a laptop with other work on it: treat
  single-percent cells as noise and the double-digit ones as signal.
• Honest per-change note against the projections in the brief: change 1
  landed above its ~1.03× projection (1.06–1.14× on call-heavy work — it
  removes an allocation per argument, not just an instruction), change 2
  landed below its ~1.09× projection (1.03–1.07× on the recursive
  workloads, flat elsewhere), and change 3 matched its ~1.18× projection on
  self-recursive code (1.11–1.14× on fib on top of the first two, 1.20× on
  loopself).

Validation

• mix format --check-formatted, mix compile --warnings-as-errors,
  mix docs --warnings-as-errors, mix dialyzer (0 errors).
• mix test --include slow --include lua53 --include differential: 2811
  passed, 16 skipped.
• Website tests: 54 passed, format clean.
• Peephole differential corpus extended with 18 programs covering the new
  opcodes; every one of them evaluates identically with peephole: false
  and peephole: true, and the whole Lua 5.3 suite still compiles with no
  widened register file and no grown instruction stream.
• New error-fidelity battery: catchable stack overflow through fused frames,
  the depth it trips at (200 with max_call_depth: 200, unchanged), an
  uncaught overflow's rendered traceback, error() and type errors raised
  under the recursion, and debug.traceback / debug.getinfo read through
  the frames — all byte-identical to the unfused build. Each case asserts the
  program still fuses first, so it can't pass vacuously.

Re-validated after the rebase onto main (which added #416's compile-time
node-id stamping under this PR): the seven commits replayed with no
conflicts, and mix format --check-formatted, mix compile --warnings-as-errors, mix docs --warnings-as-errors, and mix dialyzer
(0 errors) are all clean. mix test --include slow --include lua53 --include differential: 2869 passed, 16 skipped — the count moved from 2811
because the rebase picks up the tests that landed with the rest of the stack.
Website: 54 passed, format clean.

[davydog187]

🤖 Automated review — Opus + Sonnet consolidated findings

Two independent AI reviewers (Opus, Sonnet) reviewed this PR. No live runtime defects: the full suite passed in a clean checkout (2813 tests), ~70 hand-built fusion divergence probes (peephole on/off) all agreed, and mkregs arity boundaries (0–8 params, 16/17-wide callees, vararg mixes, method calls) came out byte-identical to PUC-Lua. The call_self fail-closed proof held against escape attempts via debug.setupvalue/getupvalue/upvalueid. Both reviewers converged on the same MEDIUM:

🟠 MEDIUM — invariant test is missing a case for the new opcode (found by both reviewers)

test/lua/compiler/max_registers_invariant_test.exs:63-73 (and the raise at :131)

register_positions/1 gained cases for all six static-arity call opcodes but not Bytecode.op_call_self() (76), and the corpus's only recursive entry is a global function fib, which never fuses. Verified: adding local function fib(n) ... return fib(n-1)+fib(n-2) ... end to @corpus → ** (RuntimeError) register_positions/1 is missing a case for opcode 76. So the one opcode this PR's fusion emits has zero coverage in the invariant walker that exists precisely to catch register undercounts — and it's a latent test bomb for the next corpus edit.

🟡 LOW

• lib/lua/compiler/peephole.ex:346-356 (callee_only_upvalue_block?/3) — the child-side escape analysis models only {:set_upvalue, i, _}/{:get_upvalue, dest, i} for the self index; set_field_upvalue/get_field_upvalue on the same index fall through the catch-all, so the proof stops failing closed there. Verified: f.x = 1 inside f compiles to set_field_upvalue on the self index and still fuses. Unexploitable today (function metatables unsupported; both builds raise identically), but it goes live the moment they are. Cheap to close while the proof is fresh.
• lib/lua/compiler/peephole.ex:333-338 — no refusal fixture where a closure declared inside the fused function captures the function's own self-upvalue (local function f(n) local function g() return f end … end). The guard reads correct but is unexercised.
• lib/lua/compiler/peephole.ex:289-293 (scratch_confined?/3) — fuse_self_calls/2 is the only part of the pass that runs when the parent contains_goto?, and the forward scan doesn't model a backward goto re-entering above the binding. No fusing goto-parent shape could be constructed (3 attempts, fused=0 each) — robustness/doc gap, not a live bug.
• lib/lua/compiler/peephole.ex:156-158 — the pre-fusion child list is passed to recompute_max_registers/3 while the post-fusion list is stored. Harmless (only upvalue_descriptors are read) but reads as an accident; reads?/3 also has no :call_self clause (safe catch-all — worth a comment so a future reorder doesn't quietly rely on it).
• Maintainability: the compiled-closure call prologue is now copy-pasted 9× (2 generic + 6 static-arity + call_self), and two copies already drift (%{callee_proto | varargs: []} inlined vs setup_vararg_proto/4 — currently equivalent). A call-protocol change now has 9 edit sites. Will be filed as a follow-up issue rather than churned here.

Notes

• Changes 1–2 (one-allocation mkregs, static-arity opcodes) are unconditional, so the peephole differential doesn't cover them at all; they were verified manually against PUC-Lua as above.
• Stack order: this PR targets perf/codegen-peephole, so compiler: key scope resolution by node id and dedupe upvalue descriptors #400 → vm: cut per-call and per-iteration overhead in the dispatcher #401 → compiler: peephole pass with fused and constant opcodes #403 must land first. Note compiler: peephole pass with fused and constant opcodes #403's review found a HIGH miscompile in the shared peephole pass (loop-exit liveness) — see that PR.

Fixes for the MEDIUM and the first two LOWs will be pushed to this branch shortly.

[davydog187]

Pushed 2b2a07c addressing the review findings. The escape-analysis proof now fails closed on get_field_upvalue / set_field_upvalue against the self index — field accesses through the name fuse via rule 3 before the self-call analysis runs, so they never surface as a get_upvalue and were falling through the catch-all (verified: local function f(n) if n == 99 then f.x = 1 ... end return f(n-1) end fused before, refuses now; unexploitable today since function indexing raises identically on both builds, but the proof should never rely on that). Added @refused fixtures for both field shapes plus a closure-declared-inside-the-body case that exercises the previously untested captures_upvalue?/2 guard. The max-registers invariant walker gained its missing op_call_self case (args at base+1..base+arg_count, result at base) along with a self-recursive local function corpus entry — confirmed opcode 76 now actually reaches the walker, where the old global-fib entry never fused. Also passed the post-fusion prototype list to recompute_max_registers/3 and documented that :call_self deliberately rides reads?/3's conservative default. Full suite green: 2796 passed, 0 failures.