Skip to content

Add secure Personal OpenClaw copilot glasses sessions - #66

Draft
iJaack wants to merge 3 commits into
Intent-Lab:mainfrom
iJaack:agent/visionclaw-dat-0.8-realtime
Draft

Add secure Personal OpenClaw copilot glasses sessions#66
iJaack wants to merge 3 commits into
Intent-Lab:mainfrom
iJaack:agent/visionclaw-dat-0.8-realtime

Conversation

@iJaack

@iJaack iJaack commented Jul 23, 2026

Copy link
Copy Markdown

What

  • Upgrade the iOS sample to Meta Wearables DAT 0.8 and harden glasses connection recovery.
  • Add a dedicated glasses-session experience with Ray-Ban mic/speaker routing, low-latency Gemini Live vision, uninterrupted queued speech, and adaptive high/medium capture throttling.
  • Add an extensible named-harness registry:
    • Eva routes to the broker-owned OpenClaw glasses agent.
    • Codex lists/reads tasks and prepares a continuation that requires a separate physical confirmation on the iPhone.
    • Meta gives an explicit native-assistant fallback instead of claiming an unsupported DAT handoff.
  • Add spoken, exact-turn, one-shot authorization for fresh photo/video capture. Negated, ambiguous, replayed, wrong-kind, late, and model-only capture attempts fail closed before the camera handler runs.
  • Add a lock-screen widget, App Intent, and deep link for opening the dedicated glasses session.
  • Add a same-Wi-Fi Mac broker with TLS pinning, explicit one-time QR pairing, signed device proofs, scoped one-shot capabilities, replay protection, durable revocation, async Eva receipts, and restart recovery.
  • Add scoped Codex task control with opaque task references, persisted confirmation receipts, revision/status rechecks, and isolated raw-Git workspaces that do not execute repository hooks or filters.

Why

The previous direct integration coupled Gemini's fast model response to slower OpenClaw execution, which caused early replies, incomplete speech, timeouts, and frozen high/medium streams. It also lacked a trustworthy boundary between model suggestions and user-authorized external actions.

This change separates the realtime glasses conversation from asynchronous harness execution, keeps media and speech responsive while work continues, and moves pairing, routing scope, replay protection, Codex confirmation, and capture authorization into app/broker-owned policy.

Security and reliability

  • No OpenClaw, Codex, Gemini, or broker credentials are placed in pairing data, Bonjour, runtime records, logs, model context, or phone responses.
  • Local broker administration requires loopback plus a stable private 256-bit bearer credential with timing-safe comparison.
  • A broker restart converts persisted nonterminal Eva work into an explicit failed receipt so the phone cannot poll forever.
  • OpenClaw reconnect reconciliation uses monotonic terminal sequences, including recovery before the first live event.
  • High/medium glasses video is paused through tool execution and final speech; playback is allowed to drain fully instead of being cut off by backend completion.

Verification

  • iOS full simulator suite: 145 passed, 2 known Meta DAT MockDeviceKit skips, 0 failed
  • Focused media/routing suite: 41 passed
  • Focused epoch-propagation router suite: 5 passed
  • Broker Node suite: 114 passed, 0 failed
  • Independent iOS review: PASS
  • Independent broker/security and adversarial admin-auth reviews: PASS
  • Release device build: succeeded
  • Deep code-sign verification: passed
    • App: com.jaack.visionclaw
    • Widget: com.jaack.visionclaw.GlassesWidget
    • Team: R3UZ5ZH3DG
  • Property-list validation, staged diff checks, and staged secret scan: passed

Intentionally deferred

  • Remote-internet routing is documented but not enabled in this increment; the Mac is never exposed directly to the public internet.
  • DAT 0.8 does not expose a video-record command, so VisionClaw gives an explicit fallback rather than claiming a recording was created.
  • Final signed iPhone installation, pairing, and physical glasses audio/camera acceptance are tracked as post-PR hardware checks.

DareDev256 and others added 2 commits July 11, 2026 20:19
The pinned DAT SDK 0.4.0 only supports Ray-Ban Meta firmware up to ~V20-V22
(per Meta's Version-Dependencies matrix). Newer glasses (e.g. the 2026 frames
running V125-generation firmware, DAT runtime 0.7.0.10.0) are never discovered:
registration succeeds and audio works, but devicesStream stays empty,
AutoDeviceSelector never emits, and requestPermission(.camera) throws .noDevice.
Reported widely (issues #200, #207, #201).

This migrates the app to DAT SDK 0.7.0 and its session-based device model:

- pbxproj: meta-wearables-dat-ios 0.4.0 -> 0.7.0
- StreamSessionViewModel: migrate StreamSession -> wearables.createSession(
  deviceSelector:) -> DeviceSession; wait for session .started before
  session.addStream(config:) -> MWDATCamera.Stream (addStream returns nil if the
  session hasn't connected yet, matching Meta's own 0.7.0 sample); remap the
  renamed StreamState/StreamError/StreamConfiguration types
- WearablesViewModel: request camera permission as soon as registration completes
  (the device does not appear in devicesStream until a permission is granted via
  Meta AI; the stock flow only requested it behind the Start-streaming button,
  which is disabled until a device appears -- a deadlock)
- Info.plist: add the 0.7.0 discovery keys from Meta's sample -- bluetooth-central
  + processing background modes, NSBonjourServices (_bonjour._tcp),
  NSLocalNetworkUsageDescription

Tested live: glasses camera POV streams end-to-end on a Ray-Ban Meta (V125-gen
firmware, iOS 26.5).
@iJaack
iJaack marked this pull request as ready for review July 23, 2026 21:35
@iJaack iJaack changed the title iOS: upgrade to DAT 0.8 and harden Gemini/OpenClaw realtime sessions Add secure Personal OpenClaw copilot glasses sessions Jul 24, 2026
@iJaack
iJaack marked this pull request as draft July 24, 2026 14:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants