Skip to content

feat(js): add VSR framing to the Node SDK - #3763

Merged
spetz merged 15 commits into
masterfrom
vsr_ts_sdk
Jul 30, 2026
Merged

feat(js): add VSR framing to the Node SDK#3763
spetz merged 15 commits into
masterfrom
vsr_ts_sdk

Conversation

@spetz

@spetz spetz commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

The Node SDK only supported classic binary framing, preventing it
from connecting to VSR servers.

Add runtime-selectable VSR framing with registration, session
recovery, request sequencing, bounded response decoding, transient
retries, leader redirection, and consumer-group support.

Keep the existing sendBinaryRequest(code, payload) API. Known commands
derive their VSR operation from the protocol registry. Codes unknown
to the SDK are sent as non-replicated requests with the command code
stored in RequestHeader.reserved, allowing the server to handle an
extension or return the authoritative error. Session-control commands
remain blocked before I/O.

@github-actions github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Jul 28, 2026
@spetz spetz added sdk Change related to sdk (client) API javascript Pull requests that update javascript code labels Jul 28, 2026
@spetz
spetz requested review from T1B0 and hubcio July 28, 2026 06:00
@codecov

codecov Bot commented Jul 28, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.00621% with 24 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.51%. Comparing base (0fa657a) to head (6cc6da2).

Files with missing lines Patch % Lines
foreign/node/src/client/client.connection.ts 97.37% 4 Missing and 3 partials ⚠️
foreign/node/src/client/client.socket.ts 98.07% 7 Missing ⚠️
foreign/node/src/client/client.frame.ts 98.23% 4 Missing ⚠️
foreign/node/src/wire/error.code.ts 70.00% 3 Missing ⚠️
foreign/node/src/client/client.ts 94.87% 2 Missing ⚠️
foreign/node/src/stream/consumer-stream.ts 87.50% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master    #3763      +/-   ##
============================================
+ Coverage     75.12%   75.51%   +0.39%     
  Complexity      969      969              
============================================
  Files          1307     1317      +10     
  Lines        152751   154895    +2144     
  Branches     128184   128707     +523     
============================================
+ Hits         114756   116971    +2215     
+ Misses        34465    34295     -170     
- Partials       3530     3629      +99     
Components Coverage Δ
Rust Core 75.50% <ø> (-0.04%) ⬇️
Java SDK 62.71% <ø> (ø)
C# SDK 71.13% <ø> (-1.14%) ⬇️
Python SDK 92.27% <ø> (ø)
PHP SDK 84.52% <ø> (ø)
Node SDK 95.55% <99.00%> (+3.30%) ⬆️
Go SDK 43.08% <ø> (ø)
Files with missing lines Coverage Δ
foreign/node/src/client/client.config.ts 100.00% <100.00%> (ø)
foreign/node/src/wire/command-set.ts 100.00% <100.00%> (ø)
foreign/node/src/wire/command.code.ts 100.00% <100.00%> (ø)
foreign/node/src/wire/command.utils.ts 100.00% <100.00%> (ø)
...de/src/wire/consumer-group/delete-group.command.ts 100.00% <100.00%> (ø)
.../node/src/wire/consumer-group/get-group.command.ts 100.00% <100.00%> (ø)
...oreign/node/src/wire/consumer-group/group.utils.ts 100.00% <100.00%> (ø)
foreign/node/src/wire/consumer-group/index.ts 100.00% <100.00%> (ø)
...node/src/wire/consumer-group/join-group.command.ts 100.00% <100.00%> (ø)
...ode/src/wire/consumer-group/leave-group.command.ts 100.00% <100.00%> (ø)
... and 20 more

... and 45 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@spetz
spetz requested a review from Copilot July 28, 2026 19:30

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Adds runtime-selectable VSR framing support to the Node SDK so it can connect to VSR servers while keeping the existing sendBinaryRequest(code, payload) API.

Changes:

  • Introduces VSR wire support (headers, operations, namespace routing, register handshake, session state, and reply decoding) plus protocol parity checks.
  • Extends the raw client transport to support both classic and VSR framing with bounded frame extraction, retries, and leader redirection.
  • Adds consumer-group support in VSR polling (SyncGroup) and expands unit/e2e/CI coverage (including a VSR e2e task).

Reviewed changes

Copilot reviewed 50 out of 51 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
foreign/node/src/wire/vsr/vsr.test.ts Adds unit coverage for VSR framing behavior from the session encoder and command preparation.
foreign/node/src/wire/vsr/session.ts Implements consensus session state for clientId/session binding and request id sequencing.
foreign/node/src/wire/vsr/session.test.ts Tests session binding, monotonic request ids, exhaustion behavior, and re-arming registration.
foreign/node/src/wire/vsr/reply.ts Implements VSR reply/eviction decoding and result-section stripping with typed errors.
foreign/node/src/wire/vsr/reply.test.ts Tests reply decoding, truncation handling, and eviction mapping.
foreign/node/src/wire/vsr/register.ts Implements VSR register/login request payloads and response deserialization.
foreign/node/src/wire/vsr/register.test.ts Tests register payload encoding/decoding and truncation paths.
foreign/node/src/wire/vsr/operation.ts Adds command-code → VSR operation mapping plus operation classification helpers.
foreign/node/src/wire/vsr/operation.test.ts Tests operation mapping/classification and known-operation filtering.
foreign/node/src/wire/vsr/namespace.ts Implements routing namespace derivation (peeking payloads for partition-plane ops).
foreign/node/src/wire/vsr/namespace.test.ts Tests namespace routing rules and payload peek validation/error paths.
foreign/node/src/wire/vsr/index.ts Provides VSR session encoder, response decode adapter, and legacy-login preparation for VSR.
foreign/node/src/wire/vsr/header.ts Defines VSR header layout/constants and request header encoding helpers.
foreign/node/src/wire/vsr/header.test.ts Tests header encoding at canonical offsets and maximum-width fields.
foreign/node/src/wire/topic/topic.utils.ts Fixes base topic field offsets to match updated wire layout.
foreign/node/src/wire/topic/topic.utils.test.ts Adds coverage for the updated topic wire layout offsets.
foreign/node/src/wire/message/poll-messages.command.ts Adds VSR consumer-group polling flow using SyncGroup and local cursoring.
foreign/node/src/wire/message/poll-messages.command.test.ts Tests VSR consumer-group polling: assignment reuse, resync, and error paths.
foreign/node/src/wire/error.utils.ts Introduces ResponseError with structured commandCode/errorCode.
foreign/node/src/wire/consumer-group/sync-group.command.ts Adds SyncGroup wire command and assignment deserialization.
foreign/node/src/wire/consumer-group/sync-group.command.test.ts Tests SyncGroup serialization/deserialization and truncation/trailing validation.
foreign/node/src/wire/consumer-group/index.ts Exposes SyncGroup from the consumer-group API surface.
foreign/node/src/wire/consumer-group/group.utils.ts Adds TargetGroup type to share stream/topic/group identifiers.
foreign/node/src/wire/command.code.ts Extends command registry with SyncGroup and offset-v2 command codes.
foreign/node/src/wire/command-set.ts Wires SyncGroup into API and keeps raw sendBinaryRequest guard behavior.
foreign/node/src/wire/command-set.test.ts Updates tests for updated sendCommand signature and custom code forwarding.
foreign/node/src/e2e/test-client.utils.ts Allows selecting protocol via env var for e2e runs.
foreign/node/src/e2e/tcp.raw.e2e.ts Expands raw e2e tests for VSR vendor codes and request-id gap behavior.
foreign/node/src/e2e/tcp.consumer-stream.e2e.ts Adjusts partitioning strategy for VSR vs classic to ensure routable partitions.
foreign/node/src/e2e/tcp.consumer-group.e2e.ts Updates message sending strategy for VSR-compatible partition selection.
foreign/node/src/e2e/tcp.cluster.e2e.ts Adds expected metadata variant for VSR path.
foreign/node/src/client/index.ts Exposes client config normalization in the public client exports.
foreign/node/src/client/client.type.ts Adds protocol/options types and config knobs for response frame size and send options.
foreign/node/src/client/client.ts Normalizes client config and refactors pooled/single client providers.
foreign/node/src/client/client.socket.ts Implements classic + VSR send pipeline, retries, leader redirection, and session lifecycle.
foreign/node/src/client/client.socket.test.ts Adds extensive VSR client socket tests (transients, eviction, redirect, concurrency).
foreign/node/src/client/client.frame.ts Adds protocol-aware frame extraction with max-size enforcement.
foreign/node/src/client/client.frame.test.ts Tests frame extraction for classic/VSR fragmentation/coalescing and size limits.
foreign/node/src/client/client.connection.ts Refactors connection lifecycle, adds redirect/abort, and uses new frame extraction.
foreign/node/src/client/client.connection.test.ts Tests connection attempt sharing, frame buffering, and shutdown error suppression.
foreign/node/src/client/client.config.ts Adds config normalization/validation for protocol and response frame limits.
foreign/node/src/client/client.config.test.ts Tests config normalization (protocol defaults, VSR constraints, size validation).
foreign/node/src/bdd/raw.ts Keeps BDD raw steps compatible with unchanged sendBinaryRequest signature.
foreign/node/scripts/check-vsr-protocol.mjs Adds build-time parity check against Rust protocol sources (codes/ops/layout/version).
foreign/node/package.json Bumps package version and adds protocol check + VSR e2e script + lint quoting fix.
foreign/node/README.md Documents selecting VSR protocol, constraints, retries, and max frame size config knob.
codecov.yml Splits Node coverage flagging by task (unit/e2e/e2e-vsr).
.github/workflows/_test.yml Updates Codecov upload logic and flags to include new e2e-vsr task.
.github/config/components.yml Adds e2e-vsr to sdk-node CI task list.
.github/actions/node-npm/pre-merge/action.yml Adds VSR server start/stop and VSR e2e run into the pre-merge action.
Files not reviewed (1)
  • foreign/node/package-lock.json: Generated file
Comments suppressed due to low confidence (2)

foreign/node/src/wire/vsr/namespace.ts:1

  • validateField accepts negative values and non-integers (e.g., -1, 1.5) and would allow them through to packNamespace, producing invalid namespaces (including negative bigint values). Since these fields are wire-level u32 identifiers, validation should reject values < 0 and non-integers (and ideally also reject NaN/Infinity) in addition to the upper bound check.
    foreign/node/src/wire/vsr/namespace.ts:1
  • namespaceFromConsumerOffset always throws errors tagged with COMMAND_CODE.StoreOffset, even when called for DeleteConsumerOffset, StoreOffset2, or DeleteConsumerOffset2. This makes thrown ResponseError.commandCode misleading and complicates debugging. Consider passing the originating command code into namespaceFromConsumerOffset (similar to other helpers here) and using it consistently in responseError(...).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread foreign/node/src/client/client.socket.ts
Comment thread foreign/node/src/client/client.type.ts
Comment thread foreign/node/scripts/check-vsr-protocol.mjs Outdated
Comment thread foreign/node/src/wire/message/poll-messages.command.ts Outdated
numinnex
numinnex previously approved these changes Jul 29, 2026
Comment thread foreign/node/src/client/client.socket.ts Outdated
Comment thread foreign/node/src/wire/vsr/namespace.ts
Comment thread foreign/node/src/wire/message/poll-messages.command.ts Outdated
Comment thread foreign/node/src/client/client.frame.ts Outdated
Comment thread foreign/node/src/client/client.config.ts
Comment thread foreign/node/src/client/client.frame.ts Outdated
Comment thread foreign/node/src/wire/vsr/index.ts Outdated
Comment thread foreign/node/src/client/client.connection.ts Outdated
Comment thread foreign/node/scripts/check-vsr-protocol.mjs Outdated
Comment thread foreign/node/src/wire/consumer-group/group.utils.ts
@github-actions github-actions Bot added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels Jul 30, 2026
@spetz

spetz commented Jul 30, 2026

Copy link
Copy Markdown
Contributor Author

/ready

@github-actions github-actions Bot added S-waiting-on-review PR is waiting on a reviewer and removed S-waiting-on-author PR is waiting on author response labels Jul 30, 2026
Comment thread foreign/node/src/client/client.connection.ts Outdated
Comment thread foreign/node/src/client/client.connection.ts
Comment thread foreign/node/src/client/client.connection.ts Outdated
@github-actions github-actions Bot added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels Jul 30, 2026
Comment thread foreign/node/src/client/client.connection.ts Outdated
Comment thread foreign/node/src/client/client.connection.ts
@spetz

spetz commented Jul 30, 2026

Copy link
Copy Markdown
Contributor Author

/ready

@github-actions github-actions Bot added S-waiting-on-review PR is waiting on a reviewer and removed S-waiting-on-author PR is waiting on author response labels Jul 30, 2026
@spetz
spetz merged commit 3b190c1 into master Jul 30, 2026
103 checks passed
@spetz
spetz deleted the vsr_ts_sdk branch July 30, 2026 12:32
@github-actions github-actions Bot removed the S-waiting-on-review PR is waiting on a reviewer label Jul 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

javascript Pull requests that update javascript code sdk Change related to sdk (client) API

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants