Add iOS platform: on-device agent, realtime voice, input reliability

[adamcohenhillel]

Summary

Adds iOS as a new platform subtree under ios/, bringing the on-device
OpenPhone agent (daemon + input tweaks) to parity with the existing
Android-focused runtime. Latest commit hardens the realtime voice pipeline and
input reliability.

Realtime voice loop

• The live conversation owns the session: the model calling finish_task /
  fail_task no longer tears down the mic/socket. Only a user cancel, a
  dropped socket, or the safety duration ceiling ends a session.
• Barge-in stays armed through tool execution and aborts an in-flight action
  batch when the user speaks over the agent (true interruptibility).
• A fresh screen observation is auto-delivered after each UI action and cached
  as the next turn's input; the wait tool is neutralized in the realtime
  path — together these cut per-step dead air.
• Disabled elements that still expose valid bounds are tapped by coordinate.
• Non-fatal jetsam memory limit raised to 1024MB, verified via kernel readback.

Input / device layer

• Volume-trigger gesture state machine hardened with generation-counter
  invalidation of stale hold timers and explicit phase tracking.
• App introspector input-focus refinements.

Test plan

• Squeeze trigger → speak a task → agent replies with voice and drives the UI
• Speak over the agent mid-action → it stops the batch and listens
• Confirm session persists after the model reports a subtask finished
• Extended session stays under the memory ceiling (no jetsam kill)
• Daemon auto-recovers after a crash (launchd relaunch)

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
openphone-docs	Ready	Preview, Comment	Jul 10, 2026 10:32pm

[chatgpt-codex-connector]

💡 Codex Review

OpenPhone/ios/contracts/protocol/openphone-ios-tools.json

Line 9 in 277a514

"command": "openphone.screen.get",

Accept the published command names

When a client follows this iOS tool manifest and sends {"command":"openphone.screen.get"} (or openphone.task.list, openphone.ui.tap, etc.), OPHandleRequest never matches those names—it only accepts legacy names like get_screen, list_apps, and execute_action, then returns unknown_command. Since this file is the published protocol surface for iOS, the daemon needs to register these canonical aliases or the manifest needs to advertise the actual accepted command strings.

---

OpenPhone/ios/agentd/Makefile

Line 31 in 277a514

openphone-protected-data-helper_FILES = src/main.m

Sign the protected helper with memorystatus entitlements

When the mobile daemon gets EPERM and delegates jetsam_priority_set to openphone-protected-data-helper, this target is built from the same memorystatus code but is not signed with entitlements.plist; the code path says the memlimit syscall requires com.apple.private.memorystatus, so the helper will still fail the priority/limit bump in that fallback case. Add the same _CODESIGN_FLAGS = -Sentitlements.plist (or a helper-specific entitlement file) to this target.

---

OpenPhone/ios/agentd/src/volume_trigger.xm

Line 1833 in 277a514

chmod(path.UTF8String, 0644);

Keep screenshot artifacts private

When SpringBoard services a screenshot request, this chmod makes the captured screen image 0644 under /var/mobile/Library/OpenPhone/screenshots; those files can contain messages, notifications, or other on-screen PII and should not be world-readable to other jailbreak-side processes. Use 0600 for the image artifact before returning the path.

---

OpenPhone/ios/agentd/src/main.m

Lines 20003 to 20005 in 277a514

	} else if ([command isEqualToString:@"app_input_poll"] ||
	[command isEqualToString:@"openphone.app_input.poll"]) {
	response = OPAppInputPoll(request);

Authenticate loopback app-input requests

When any other app/process on the device can connect to 127.0.0.1:27631, this branch lets it call app_input_poll with an arbitrary bundle_id and receive pending input actions, including the text being typed, or complete/spoof those requests before the injected app process does. Loopback alone is not an identity check here; add a per-install shared secret or move this bridge to a permissioned Unix socket before exposing app input over it.

---

OpenPhone/ios/agentd/src/main.m

Lines 1529 to 1531 in 277a514

	event[@"previous_hash"] = OPLastAuditHash();
	event[@"event_hash"] = OPSHA256Hex(OPCanonicalJSONData(event));
	OPAppendJSONLine(OPAuditPath(), event);

Serialize audit hash appends

When two async paths record audit events at the same time (for example a voice task and the background scheduler), both can read the same OPLastAuditHash() before either append completes, so the second event's previous_hash no longer matches the immediately preceding line and validate-agentd-store.py will reject the audit chain. Guard the previous-hash calculation and append with a shared lock or append through a single audit writer.

---

OpenPhone/ios/agentd/src/main.m

Lines 1743 to 1744 in 277a514

	int rc = sqlite3_open_v2(OPDatabasePath().UTF8String, &db,
	SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE | SQLITE_OPEN_FULLMUTEX, NULL);

Restrict the SQLite store permissions

When SQLite creates openphone.sqlite here, it uses the process umask under a 0755 store directory, so the database is typically left readable by other local jailbreak-side processes even though it stores memory/context rows and clipboard/notification-derived text. Set the db directory/file to 0700/0600 after creation (and do the same for the protected-data-helper store) before writing user data.

---

OpenPhone/ios/tools/mac/agentd/validate-on-device.sh

Line 132 in 277a514

validation_root="${OPENPHONE_VALIDATION_DIR:-$repo_root/artifacts/validation}"

Write validation artifacts under an ignored path

With the default settings, on-device validation writes pulled logs, screenshots, store snapshots, and other privacy-sensitive artifacts under ios/artifacts/validation, but this path is not ignored by the new .gitignore; running the validator leaves those files as ordinary untracked repo content that can be committed accidentally. Default this to an ignored directory or add ios/artifacts/ (or the validation subtree) to .gitignore.

---

OpenPhone/ios/shared/contracts/model-tools.json

Line 10 in 277a514

{ "tool": "swipe", "capability": "input.perform", "drives_ui": true, "arguments": { "from_x": "number", "from_y": "number", "to_x": "number", "to_y": "number", "expected_visible_change": "string" } },

Use daemon swipe argument names

The shared model-tools contract tells consumers to emit swipe arguments as from_x/from_y/to_x/to_y, but the daemon and prompt/tool definitions only read start_x/start_y/end_x/end_y; a model or Android consumer generated from this shared contract will have every swipe rejected as missing_swipe_coordinates. Update this contract to the daemon's names or add a normalization layer before OPExecuteAction.

---

OpenPhone/ios/agentd/src/main.m

Line 12812 in 277a514

NSDictionary *typed = OPPerformHIDTypeText(text);

Check focus before dispatching blind typing

When the app-input bridge does not handle type_text and the screen has no focused editable field, this still dispatches the user's text via HID before returning action.denied.no_focused_field. In that scenario the code is explicitly trying not to trust blind typing, but the text may already have gone to a hidden or incorrect responder; return the denial before calling OPPerformHIDTypeText when hasFocusedField is false.

---

OpenPhone/ios/agentd/src/main.m

Lines 16329 to 16330 in 277a514

	NSArray *approved = [request[@"approved_capabilities"] isKindOfClass:[NSArray class]]
	? request[@"approved_capabilities"] : OPFullYoloCapabilities();

Preserve stored task capabilities on attach

When a caller starts a task with restricted approved_capabilities and later runs it by task_id, this default ignores the capabilities stored on the task file because the run request usually omits approved_capabilities, giving the model OPFullYoloCapabilities() instead. That bypasses reviewed/limited task scopes; after loading an adopted task, use its stored approved_capabilities before falling back to full YOLO.

---

OpenPhone/ios/agentd/src/main.m

Line 11086 in 277a514

chmod(OPNotificationLogPath().UTF8String, 0644);

Keep notification previews private

When notifications are ingested, the rolling log stores each bundle, title, subtitle, body preview, and thread id, then makes the file 0644 under /var/mobile/Library/OpenPhone. Those previews can contain message or 2FA content; keep this log 0600 like other protected JSON files so unrelated local processes cannot read it.

---

OpenPhone/ios/agentd/src/volume_trigger.xm

Line 5895 in 277a514

if (!OPVTWriteJSONDictionary(OPVTClipboardResponsePath, response ?: @{})) {

Keep clipboard bridge responses private

When clipboard_read is serviced by SpringBoard, response contains the full pasteboard string and is written to clipboard-response.json through OPVTWriteJSONDictionary, which chmods JSON outputs to 0644; the daemon also leaves the response file behind after reading it. This leaves stale clipboard contents readable to other local processes, so write this bridge response with 0600 and remove it after consumption.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".