Skip to content

feat(mediaplayer): add RIST live streaming with loss recovery + encryption#853

Open
towneh wants to merge 4 commits into
BasisVR:developerfrom
towneh:feat/rist-on-mediaplayer
Open

feat(mediaplayer): add RIST live streaming with loss recovery + encryption#853
towneh wants to merge 4 commits into
BasisVR:developerfrom
towneh:feat/rist-on-mediaplayer

Conversation

@towneh

@towneh towneh commented Jun 10, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds RIST (Reliable Internet Stream Transport) as a receive-only live-ingest transport for the media player — broadcast-grade loss recovery (ARQ) plus optional content encryption (Main-Profile PSK-AES) for live shows. This lands the prototype discussed in #848.

It's additive and low-blast-radius:

  • Sits behind a BASIS_WITH_RIST CMake option that is off by default, so existing builds are byte-for-byte unchanged and link no new dependencies.
  • Reuses the existing pipeline end to end — librist hands recovered MPEG-TS to the same basis_ts_run demuxer, the same OS decode / zero-copy texture path, and the same exponential-backoff reconnect loop. No new framework surface, no new events.
  • rist:// is admitted through the existing BasisMediaPlayerSecurity allowlist (validated at the boundary), and an optional Options["buffer"] (ms) is folded into the URL query that librist parses.

librist itself is a third-party native dependency built from source in CI (v0.2.11, its bundled mbedTLS linked into a single static), not committed — the shipping answer to the "how should the first native dep be shipped" question from #848.

Commits (logical, reviewable):

  1. feat(mediaplayer): RIST live-ingest transport (native + CMake) — the receiver module (basis_rist.c/.h), rist:// scheme + query handling in basis_url, the run_rist dispatch, and the CMake wiring.
  2. feat(mediaplayer): accept rist:// URLs in the player — the security allowlist entry and the Options["buffer"] → query fold.
  3. build(mediaplayer): build librist from source for RISTbuild-librist.{ps1,sh} + the media-native.yml workflow.
  4. docs(mediaplayer): record librist in README and THIRD_PARTY_NOTICES — BSD-2-Clause licensing + the optional-build note.

Design detail (technical + business specs) is linked from #848.

basis_rist_gh.mp4

Demo: AES-128 rist:// stream decoding live in the editor — clip attached on #848

Required checks

All boxes below must be ticked before this PR can merge. If a check is genuinely N/A, tick it anyway and explain under Notes.

  • Tested — I built and ran this locally. The change works in the editor and (where relevant) in a built player.
  • Transform access is combined and limited — In hot paths, transform reads/writes go through TransformAccessArray or are otherwise batched. I have not added per-frame transform.position / transform.rotation / transform.localPosition calls inside loops. Whenever I need both position and rotation, I use the combined APIs — SetPositionAndRotation / SetLocalPositionAndRotation for writes, GetPositionAndRotation / GetLocalPositionAndRotation for reads — instead of two separate property accesses; the combined call does one local-to-world matrix traversal instead of two.
  • Addressables used for asset/memory loading — Any new asset loads go through Addressables. No new Resources.Load, no direct asset references that pull large content into memory on scene load.
  • No new GetComponent / AddComponent where avoidable — Where unavoidable, the result is cached on a field, and any GetComponent<T> is replaced with TryGetComponent<T>(out var x) — bare GetComponent will be denied. TryGetComponent is the modern API (Unity 2019.2+) and skips the Editor-only GC allocation GetComponent causes when a component is missing: Unity wraps the null return in a managed "fake null" object so its overloaded == operator can still detect destroyed C++ objects, and constructing that wrapper allocates; TryGetComponent returns a bool plus out parameter and never builds the wrapper. None of these calls run inside Update, LateUpdate, FixedUpdate, jobs, or other per-frame code paths.
  • Per-frame work is scheduled through BasisEventDriver — Any new per-frame work hooks into BasisEventDriver rather than adding standalone Update / LateUpdate / FixedUpdate callbacks on a MonoBehaviour.
  • Anything added to BasisEventDriver is bulletproof, or guarded by try/catchBasisEventDriver runs the single per-frame tick that drives the whole framework (network apply, local player sim, blendshapes, JigglePhysics, nameplates, and more) as one sequential chain. An unhandled exception anywhere in that chain aborts the rest of the tick, so every step after the throwing one is silently skipped for that frame. New work added to the driver must either be guaranteed not to throw, or be wrapped in a try/catch that contains the failure and surfaces it through BasisDebug — logged once / rate-limited, never every frame (see the existing HVRBasisBuiltInAddresses.Simulate() guard for the pattern). Expect this to be scrutinized closely in review.
  • Considered jobification — I asked whether this work can be moved to a Unity Job (Burst-compiled where possible). If it can, it is. If it cannot, the reason is in Notes.
  • No needless { get; set; } properties or access lockdowns — Public fields are fine; Basis is meant to be read and modified freely, so don't wall things off private/internal without a real reason. Don't wrap a field in { get; set; } when the accessors do nothing — property accessors have a real performance cost vs direct field access, and the lead maintainer prefers plain fields (or a method / setter-only property when only the setter needs logic) over a noop-getter pair. For .Instance singletons, callers reassigning Type.Instance is allowed; if that would break your code, log a warning or throw — don't block the assignment. Locking down access is not your call.
  • Camera access goes through BasisLocalCameraDriver — Code that needs the local camera (transform, projection, rig data, etc.) pulls it from BasisLocalCameraDriver rather than looking one up itself. Don't roll a separate camera discovery path.
  • Logging uses BasisDebug — All new logging calls go through BasisDebug.Log / BasisDebug.LogWarning / BasisDebug.LogError (with an appropriate LogTag) instead of UnityEngine.Debug.Log / Debug.LogWarning / Debug.LogError. BasisDebug routes through Basis's tagged, color-coded logger and respects the project-wide LoggingDisabled toggle so logging can be killed at runtime; bare Debug.Log calls bypass that and will be denied.
  • No scene-wide discovery for dependencies — New code is architected so it does not need FindObjectOfType / FindObjectsOfType / GameObject.Find / FindGameObjectsWithTag to locate what it depends on. References are wired in — registered through an existing manager/driver, injected at init, or passed in by the caller — rather than discovered by scanning the scene at runtime. If a scene scan is genuinely unavoidable, justify it under Notes.
  • No allocations in hot paths — Per-frame code (Update / LateUpdate / FixedUpdate, simulation loops, jobs, anything called once per frame or more) does not allocate. No new on reference types, no LINQ, no string concatenation/interpolation, no boxing, no foreach over interface-typed collections. Allocate once at init and reuse the buffer.
  • No debugging in hot paths — No log calls of any kind on per-frame paths, including BasisDebug. Hot-path logging floods the console and incurs cost on every frame regardless of whether the message is filtered out downstream. If a hot-path log is needed while iterating, gate it behind #if UNITY_EDITOR and remove (or leave gated) before merge.
  • Hot-path collection access is optimized — Cache .Count (lists) / .Length (arrays) into a local int before the loop instead of re-reading the property each iteration. Prefer T[] (with a separate length int when the array is over-sized) over List<T> where the data is hot — Unity's mono BCL doesn't expose CollectionsMarshal.AsSpan(List<T>), so a list can't be fed into Span<T> / unsafe paths cleanly. Where the perf justifies it, drop into Span<T> / ref locals / Unsafe.As / unsafe pointer code to skip bounds checks and copies, and call out the invariants you're relying on under Notes so reviewers can sanity-check them.

Testing details

Tick the platforms you actually tested on. Leave the rest unticked — these are informational and do not block merge.

  • Windows
  • Linux
  • Android
  • iOS
  • macOS

Input / control mode coverage:

  • Tested in VR (note headset under Notes)
  • Tested in desktop / non-VR mode
  • Tested with phone controls (mobile touch input)
  • N/A — change does not touch player/XR/input code

Where applicable, confirm these flows still work after your changes:

  • Hot-switching (desktop ↔ VR mode swap at runtime)
  • Avatar swapping
  • Server swapping (joining / leaving / changing servers)
  • N/A — change does not touch any of the above

Notes

What was tested. Validated end-to-end in the Unity editor on Windows against a live librist broadcaster over the public internet, on an optimized Release build of the plugin (the shape that ships, no debug info):

  • Both an unencrypted rist://host:port stream and a PSK-AES-128 rist://host:port?secret=…&aes-type=128 stream play the full pipeline — RIST receive → (decrypt) → MPEG-TS demux → Media Foundation H.264 decode → zero-copy texture, with audio.
  • A mid-stream reload exercises the reconnect loop and recovers cleanly (player diagnostics: zero dropped/skipped frames, ~1.8 s reconnect time-to-first-frame, ~240 ms presentation latency — sub-second, fine for live).

Android (Quest) is the natural next platform but is not device-tested yet, so it's left unticked. The librist Android static already builds in CI and the demux/decode path is shared, so it should be mostly an on-device build + run; I'd rather leave it honest than fake-green it.

Why most required checks are N/A. This change is a native-C transport plus a small init-path C# change — it adds no per-frame/Unity-runtime work, so most of the perf checklist doesn't apply: no transforms, no Addressables/asset loads, no GetComponent/AddComponent, nothing added to BasisEventDriver, no camera access, no scene-wide discovery, no new C# logging (native failures surface through the existing sink->on_error → player error path). The two that genuinely bite, and hold:

  • No hot-path allocations — the native side allocates its TS ring buffer once at basis_rist_open and reuses it; basis_rist_read does no per-read allocation. The C# Options["buffer"] fold runs once per LoadUrl, not per frame.
  • Jobification — N/A; the receive/recovery work runs on librist's own internal threads (UDP/ARQ/jitter), and demux runs on the existing media thread, not the Unity job system.

Default builds are unaffected. With BASIS_WITH_RIST off (the default), basis_rist.c compiles only a graceful "not built" stub and the plugin links exactly as today — no librist, no new system libs.

towneh added 4 commits June 11, 2026 00:29
Add RIST (Reliable Internet Stream Transport) as a receive-only live transport, behind a BASIS_WITH_RIST CMake option that is OFF by default so existing builds are unaffected and link no new dependencies.

librist (Main Profile) owns the UDP sockets, ARQ recovery, jitter buffer and optional PSK-AES; basis_rist.c buffers the recovered MPEG-TS through a lock-guarded ring and exposes it as a basis_read_fn, so basis_ts_run demuxes RIST exactly like the existing TCP/HTTP byte sources and rides the same reconnect loop. basis_url gains the rist:// scheme and captures the query (secret/aes-type/buffer) that librist parses.

The librist static is built separately (see the build commit); third_party/ holds only .gitkeep scaffolding and a build README.
Admit rist:// through the existing BasisMediaPlayerSecurity allowlist (boundary validation, no new code path), and fold an optional Options["buffer"] (milliseconds) into the URL query so librist sees the receive-buffer depth. Non-RIST schemes are untouched.
librist is built from source rather than committed: build-librist.{ps1,sh} clone the pinned tag (v0.2.11) and build a single static (its bundled mbedTLS linked in) into third_party/<rid>/, run by CI and locally alike. media-native.yml builds the static per platform (Windows x64, Android arm64) and uploads it as an artifact. The statics stay gitignored.
Note the optional RIST build and librist's BSD-2-Clause licensing (and its vendored mbedTLS) in the package README and third-party notices.
@towneh towneh changed the title feat(mediaplayer): RIST live-ingest transport (behind BASIS_WITH_RIST) feat(mediaplayer): add RIST live streaming with loss recovery + encryption Jun 10, 2026
@towneh towneh requested a review from dooly123 June 10, 2026 23:46
@towneh towneh added the enhancement New feature or request label Jun 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant