From 9f2697f5ca7afef35e16db1c42a14a02c7f5de24 Mon Sep 17 00:00:00 2001 From: Jacob Fu <141651335+FuJacob@users.noreply.github.com> Date: Fri, 17 Jul 2026 21:01:18 -0700 Subject: [PATCH 1/3] Sync documentation --- .claude/CLAUDE.md | 7 +- .gitignore | 1 - .internal/README.md | 85 ++ .../context-privacy-and-permissions.md | 252 +++++ .../architecture/focus-and-accessibility.md | 232 +++++ .../architecture/inference-and-prompting.md | 239 +++++ .internal/architecture/input-and-insertion.md | 199 ++++ .../architecture/lifecycle-and-composition.md | 195 ++++ .../presentation-and-sibling-features.md | 241 +++++ .internal/architecture/suggestion-pipeline.md | 265 ++++++ .internal/interview-prep/README.md | 802 ++++++++++++++++ .../hyperwrite-reliability-translation.md | 878 ++++++++++++++++++ .../interview-prep/technical-question-bank.md | 877 +++++++++++++++++ AGENTS.md | 23 +- ARCHITECTURE.md | 31 +- 15 files changed, 4308 insertions(+), 19 deletions(-) create mode 100644 .internal/README.md create mode 100644 .internal/architecture/context-privacy-and-permissions.md create mode 100644 .internal/architecture/focus-and-accessibility.md create mode 100644 .internal/architecture/inference-and-prompting.md create mode 100644 .internal/architecture/input-and-insertion.md create mode 100644 .internal/architecture/lifecycle-and-composition.md create mode 100644 .internal/architecture/presentation-and-sibling-features.md create mode 100644 .internal/architecture/suggestion-pipeline.md create mode 100644 .internal/interview-prep/README.md create mode 100644 .internal/interview-prep/hyperwrite-reliability-translation.md create mode 100644 .internal/interview-prep/technical-question-bank.md diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index 596a0529..6f25409e 100644 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -2,6 +2,7 @@ @../AGENTS.md -`AGENTS.md` is the canonical project instruction file. Keep shared architecture, workflow, safety, -debugging, validation, and documentation rules there so Claude Code and other coding agents receive -the same guidance without duplicated copies drifting apart. +`AGENTS.md` is the canonical coding-agent instruction file. Keep shared workflow, safety, debugging, +validation, and architecture rules there so Claude Code and other agents receive the same guidance +without duplicated copies drifting apart. `ARCHITECTURE.md` is the ten-minute system map, while +`.internal/` contains the committed deep-dive and interview-preparation documentation. diff --git a/.gitignore b/.gitignore index 3e5b784f..9840988a 100644 --- a/.gitignore +++ b/.gitignore @@ -42,4 +42,3 @@ posts.txt .rocketride/ .agents -.internal diff --git a/.internal/README.md b/.internal/README.md new file mode 100644 index 00000000..6534bbb7 --- /dev/null +++ b/.internal/README.md @@ -0,0 +1,85 @@ +# Cotabby Internal Documentation + +This directory holds detailed maintainer and onboarding notes that are intentionally more exhaustive +than the repository's root architecture map. Start with [ARCHITECTURE.md](../ARCHITECTURE.md) for the +ten-minute system tour, then use the guides below to follow one responsibility through its owners, +data flow, invariants, and failure modes. + +The .internal directory is committed maintainer documentation. It may discuss implementation debt and +interview preparation more candidly and deeply than public product documentation, but it must not +contain credentials, private user data, or claims that cannot be verified from the repository. The +root architecture map remains self-contained and does not require these guides to be useful. + +## Architecture Guides + +| Guide | Question it answers | +| --- | --- | +| [Lifecycle and Composition](architecture/lifecycle-and-composition.md) | Who constructs, starts, retains, and stops app-lifetime objects? | +| [Suggestion Pipeline](architecture/suggestion-pipeline.md) | How does one input event become a streamed, visible, and accepted suggestion? | +| [Focus and Accessibility](architecture/focus-and-accessibility.md) | How does Cotabby resolve a safe field, bounded text, identity, and caret geometry? | +| [Input and Insertion](architecture/input-and-insertion.md) | Which global events are observed or consumed, and how is text committed safely? | +| [Inference and Prompting](architecture/inference-and-prompting.md) | How do Apple, llama, and endpoint requests differ while sharing one output contract? | +| [Context, Privacy, and Permissions](architecture/context-privacy-and-permissions.md) | What data can be acquired, where is it bounded, and when can it leave the Mac? | +| [Presentation and Sibling Features](architecture/presentation-and-sibling-features.md) | How do overlays, emoji, macros, settings, and onboarding share presentation infrastructure? | + +## Interview Preparation + +The interview-prep layer reuses the architecture guides rather than restating them: + +| Guide | Purpose | +| --- | --- | +| [Untimed Study Path](interview-prep/README.md) | Master the repository through exact source files, symbols, active-recall checkpoints, and six complete execution traces. | +| [Technical Decision Question Bank](interview-prep/technical-question-bank.md) | Practice difficult architecture questions with strong answers, tradeoffs, source trails, and honest limitations. | +| [HyperWrite Reliability Translation](interview-prep/hyperwrite-reliability-translation.md) | Apply Cotabby's lessons to inspecting, scoping, and hardening the HyperWrite Mac prototype into a measurable alpha. | + +## Suggested Reading Routes + +For the product loop and interview-level architecture discussion: + +1. Root ARCHITECTURE.md +2. Lifecycle and Composition +3. Suggestion Pipeline +4. Inference and Prompting +5. Context, Privacy, and Permissions + +For a field compatibility or ghost-positioning problem: + +1. Focus and Accessibility +2. Presentation and Sibling Features +3. Suggestion Pipeline + +For acceptance, IME, clipboard, or lost-keystroke behavior: + +1. Input and Insertion +2. Suggestion Pipeline +3. Focus and Accessibility + +For a new generation backend or context source: + +1. Inference and Prompting +2. Context, Privacy, and Permissions +3. Suggestion Pipeline +4. Lifecycle and Composition + +For the HyperWrite technical session: + +1. Untimed Study Path +2. Technical Decision Question Bank +3. HyperWrite Reliability Translation + +## Accuracy Standard + +These documents describe the current code, including uncomfortable limitations. They do not turn a +planned feature, stale comment, or desired privacy property into a shipping claim. + +When architecture changes: + +1. Verify behavior from the concrete owners and tests, not only comments or older docs. +2. Update the detailed guide whose responsibility changed. +3. Update root ARCHITECTURE.md only when the ten-minute mental model or a major invariant changes. +4. Keep source links resolvable and name the actual owner rather than only a folder. +5. Record a current limitation explicitly when implementation and desired invariant differ. + +The most important known example is secure-field acquisition: current generation and insertion fail +closed, but early bounded AX context and optional visual capture can still occur. The privacy guide +documents the exact boundary; do not simplify it to a no-capture guarantee until the code changes. diff --git a/.internal/architecture/context-privacy-and-permissions.md b/.internal/architecture/context-privacy-and-permissions.md new file mode 100644 index 00000000..12703c6e --- /dev/null +++ b/.internal/architecture/context-privacy-and-permissions.md @@ -0,0 +1,252 @@ +# Context, Privacy, and Permissions + +## Purpose + +This guide inventories what Cotabby can observe, how each context source is bounded before +generation, which macOS permissions authorize the work, and when data can leave the Mac. Privacy is +an architectural constraint rather than a marketing label: every new context source needs an owner, +a permission boundary, a lifetime, a prompt budget, and a truthful disclosure for the selected +engine. + +## Context Flow + +One request can combine several independently enabled sources: + +~~~text +focused field + -> bounded preceding and trailing AX text + -> app, window, domain, placeholder, and surface metadata + -> optional user-authored rules and extended context + -> optional relevant clipboard description + -> optional screenshot-derived OCR excerpt + -> language and prediction settings + -> sanitization and per-section budgets + -> immutable SuggestionRequest + -> selected generation engine +~~~ + +[SuggestionRequestFactory.swift](../../Cotabby/Support/Suggestion/SuggestionRequestFactory.swift) is the final +assembly boundary. It receives values already acquired by their owning services; it does not capture +the screen, read the pasteboard, query AX, or open credentials itself. + +## Required and Optional Permissions + +[PermissionManager.swift](../../Cotabby/Services/Permission/PermissionManager.swift) publishes three +TCC grant states: + +- Accessibility is required to discover the focused editable element, read bounded text and + selection state, resolve geometry, validate acceptance, and perform some insertion fallbacks. +- Input Monitoring is required to observe global keyboard input and intercept accepted keys. +- Screen Recording is optional and used only for screenshot-derived visual context. + +Core autocomplete requires Accessibility and Input Monitoring. Missing Screen Recording does not +disable text-only suggestions; visual context reports an explicit unavailable state and the rest of +the pipeline continues. + +PermissionManager refreshes when Cotabby becomes active and uses a short polling loop only while a +required grant is missing. Once the required set is granted, it avoids a permanent permission poll. + +[PermissionGuidanceController.swift](../../Cotabby/Services/Permission/PermissionGuidanceController.swift) +owns how the user is guided through System Settings. It first invokes the native request API so +macOS registers the current signed application identity, then opens or overlays the relevant privacy +pane. Views ask for guidance; they do not implement TCC flow logic. + +The production and development application identities are intentionally distinct. A permission +grant belongs to the code identity macOS sees, so changing target identity, signing, or install path +can legitimately require a new grant. + +## Secure and Unsupported Fields + +[SecureFieldDetector.swift](../../Cotabby/Support/Accessibility/SecureFieldDetector.swift) centralizes detection +signals across native and non-native AX surfaces. A detected secure field receives blocked capability, +which prevents prediction, prompt construction, presentation, acceptance, and inline emoji or macro +capture. Surface metadata and field-style probes are also skipped. + +The current acquisition boundary is less strict than the assistance boundary. FocusSnapshotResolver +reads and bounds the field value, constructs FocusedInputSnapshot, and then returns the blocked +capability with that context still attached. SuggestionAvailabilityEvaluator also calls visual +capture with capability checking disabled; its code explicitly treats secure fields like other +transient blocked states so screenshot/OCR can warm early. + +That visual excerpt cannot enter an engine request because prediction remains blocked. However, the +screenshot and OCR still occur when Screen Recording and visual context are enabled, and explicit +debug mode can write the screenshot/OCR pair to disk. Therefore the accurate current guarantee is +"secure fields are never assisted or sent to an engine," not "secure fields are never captured." +Moving the secure check ahead of AX value construction and visual eligibility would be the path to +the stronger privacy invariant. + +Read-only, incompatible, explicitly disabled, or terminal surfaces are handled by capability and +availability policy. Disabling an application also prevents expensive compatibility walks where +possible, reducing both observation and the chance of disturbing a fragile AX tree. + +## Focused Text + +[FocusSnapshotResolver.swift](../../Cotabby/Services/Focus/FocusSnapshotResolver.swift) bounds text on +both sides of the caret before publication. This prevents a large document from propagating through +Combine state, signatures, logs, or request construction. SuggestionRequestFactory applies a smaller +engine-specific prefix budget later. + +Trailing text is retained only as much as Cotabby needs to avoid duplicating content after the caret +and to validate that a suggestion still belongs at the same seam. It is not an invitation to copy an +entire document into the prompt. + +## Surface Context + +[SurfaceContextComposer.swift](../../Cotabby/Support/Context/SurfaceContextComposer.swift) turns optional +application, window, browser-domain, placeholder, role, and surface metadata into a short descriptive +section. [SurfaceContextCache.swift](../../Cotabby/Services/Focus/SurfaceContextCache.swift) avoids +re-querying focus-session invariants on every poll. + +Surface metadata helps distinguish an email reply, browser editor, chat field, or document surface +without injecting an AX tree dump. Missing attributes remain absent; Cotabby does not fabricate +semantic certainty from a generic role. + +## User Context and Rules + +Extended context and custom writing rules are explicitly authored or enabled by the user. They are +sanitized and character-budgeted as optional prompt sections. These sections condition output but do +not outrank the recent caret prefix when prompt space is scarce. + +Rules, languages, and settings are durable non-secret preferences stored through +SuggestionSettingsStore. SuggestionSettingsData groups those values by product domain in memory, +while the store preserves the established flat UserDefaults keys for compatibility. Endpoint bearer +tokens are different: they are secrets and live in Keychain. + +## Clipboard Context + +[ClipboardContextProvider.swift](../../Cotabby/Services/Context/ClipboardContextProvider.swift) +reads a fresh bounded description only when the coordinator is building context. It does not publish +or retain a continuous clipboard history. Text is normalized; images and image files become compact +metadata descriptions rather than raw pixel prompt payloads. + +[ClipboardRelevanceFilter.swift](../../Cotabby/Support/Context/ClipboardRelevanceFilter.swift) prevents an +unrelated pre-existing clipboard from entering every request. It tracks pasteboard identity and +recentness, requires meaningful token overlap with the current prefix for longer content, and pins an +accepted relevance verdict to the active field so normal typing does not make context flicker. + +[ClipboardContentDistiller.swift](../../Cotabby/Support/Context/ClipboardContentDistiller.swift) keeps short +content intact and extracts overlapping lines from longer content. If no useful line can be found, +it falls back to a bounded head rather than carrying an unbounded clipboard dump. + +Clipboard context and clipboard-based insertion are separate concerns. Disabling clipboard prompt +context does not disable the IME-safe insertion fallback, which touches the pasteboard transiently and +restores the user's representations. + +## Visual Context Lifecycle + +[VisualContextCoordinator.swift](../../Cotabby/Services/Visual/VisualContextCoordinator.swift) owns +one field-scoped visual session. On a supported focus it: + +1. Coalesces rapid focus churn so Chromium and Electron settling does not launch duplicate work. +2. Checks whether visual context is enabled and Screen Recording is granted; capability is currently + ignored here, including secure-field capability. +3. Creates a session ID tied to the focused input. +4. Runs screenshot and OCR work asynchronously. +5. Applies the excerpt only if the session is still current. +6. Keeps the result through ordinary typing in the same field. +7. Cancels and clears it when focus continuity ends. + +Status is explicit: disabled, waiting, capturing, extracting text, ready, unavailable, or failed. +Missing optional permission is therefore distinguishable from an OCR failure and from a feature the +user turned off. + +## Screenshot to OCR + +[WindowScreenshotService.swift](../../Cotabby/Services/Visual/WindowScreenshotService.swift) uses +ScreenCaptureKit to select the visible window belonging to the focused process and capture a compact +region around the input. It excludes desktop windows and converts coordinate systems at the capture +boundary. + +[ScreenTextExtractor.swift](../../Cotabby/Services/Visual/ScreenTextExtractor.swift) runs Apple Vision +OCR and carries recognition confidence per line. [OCRTextHygiene.swift](../../Cotabby/Support/Context/OCRTextHygiene.swift) +then removes low-confidence lines, replacement characters, symbol-heavy noise, likely UI chrome, +digit-corrupted prose, and lines that echo the field's own text. + +[ScreenshotContextGenerator.swift](../../Cotabby/Services/Visual/ScreenshotContextGenerator.swift) +normalizes, sanitizes, checks meaningful signal, and applies a final excerpt cap. It caches a few raw +OCR extractions by sampled pixel hash, but reruns hygiene and field-text stripping on each use. + +There is no model summarization step. The prompt receives cleaned, bounded OCR text. This avoids a +second generation, avoids summary hallucination, and lets a base completion model condition on the +visible language directly. Raw screenshots never enter the suggestion prompt. + +## Prompt Sanitization and Budgets + +[PromptContextSanitizer.swift](../../Cotabby/Support/Context/PromptContextSanitizer.swift) normalizes optional +context and caps it before rendering. BaseCompletionPromptRenderer applies per-section budgets again +when assembling the final continuation prompt. + +The two layers serve different purposes: acquisition boundaries prevent oversized values from +circulating through the app, while prompt budgets decide which already-safe sections fit in this +particular request. No context source should rely only on a last-minute total-string truncation. + +## Where Data Is Processed + +Apple Intelligence and the in-process llama engine perform suggestion generation on the Mac. The +visual pipeline also uses local ScreenCaptureKit and Vision APIs. + +The OpenAI-compatible engine sends the constructed request to the endpoint the user configured. A +loopback Ollama address stays on the Mac, a LAN address sends it across the local network, and a +public HTTPS address sends it to that remote service. Endpoint privacy classification and settings UI +must communicate that distinction. Insecure public HTTP is rejected. + +Model discovery/downloads and update checks can use the network, but those operations are separate +from sending focused text for generation. Adding any hosted generation dependency or telemetry path +requires an explicit product decision and documentation update. + +## Secrets and Persistence + +- Non-secret settings persist in UserDefaults through SuggestionSettingsStore. +- Endpoint bearer tokens persist in the user's Keychain. +- Focus snapshots, clipboard context, visual excerpts, and active requests are memory-scoped values. +- Visual OCR cache entries are small in-memory extractions bounded to a few pixel hashes. +- No normal product path maintains a document, clipboard, screenshot, or prompt history database. + +## Diagnostics Are a Privacy Mode + +Normal operation sends structured events to Apple's unified logging. Launching with +-cotabby-debug intentionally enables privacy-sensitive local artifacts for development: + +- ~/Library/Logs/Cotabby/cotabby.jsonl for structured events +- ~/Library/Logs/Cotabby/llm-io.jsonl for full prompts and completions +- ~/Desktop/cotabby-ax-dump.txt for the latest Chrome AX tree snapshot +- ~/Desktop/cotabby-debug-screenshots/ for bounded retained screenshot/OCR pairs + +The LLM-I/O records share request IDs with the main stream. Visual debug captures are retained only +under the explicit debug gate and pruned per application. Anyone collecting a bug report must treat +these artifacts as potentially containing user text and screen content. + +## Invariants + +- Secure fields never schedule generation, presentation, acceptance, or inline-command capture. +- Current AX and visual acquisition can still run before/around the secure capability block; this is + a documented privacy limitation, not a no-capture guarantee. +- Accessibility and Input Monitoring gate core autocomplete; Screen Recording remains optional. +- Every text source is bounded before request assembly. +- Optional sections are sanitized and budgeted independently. +- Clipboard context is relevance-filtered and is not a continuous history. +- A visual excerpt belongs to one field-scoped session and stale work cannot apply. +- Visual context uses cleaned OCR, never raw screenshots or model-generated summaries. +- On-device and configured-endpoint generation are disclosed as different privacy scopes. +- Endpoint secrets live in Keychain. +- Privacy-sensitive files exist only when the developer explicitly launches debug mode. + +## Failure-Oriented Reading + +- Suggestions run with a missing required grant: PermissionManager and availability evaluation. +- Password field receives a ghost, generation request, or command capture: secure detection and + focus capability. +- Requirement is that secure fields are never captured at all: current focus-value and visual-context + acquisition must be changed; the existing block applies later. +- Clipboard text appears unrelated: relevance identity, overlap, and field pinning. +- Visual context follows the previous field: session ID and focus-scoped apply guard. +- Screenshot context is noisy: OCR confidence, hygiene order, and prompt sanitizer. +- Screen Recording denial disables all suggestions: required-versus-optional permission policy. +- Public endpoint has no warning: endpoint scope classification and settings presentation. +- Prompt content appears on disk unexpectedly: CotabbyDebugOptions bootstrap and launch arguments. + +## Update This Guide When + +Update this document whenever Cotabby observes a new data source, changes a permission requirement, +changes secure-field policy, adds a persistent cache, changes endpoint behavior, writes a new debug +artifact, or alters a context budget or lifetime. diff --git a/.internal/architecture/focus-and-accessibility.md b/.internal/architecture/focus-and-accessibility.md new file mode 100644 index 00000000..28c6af04 --- /dev/null +++ b/.internal/architecture/focus-and-accessibility.md @@ -0,0 +1,232 @@ +# Focus and Accessibility + +## Purpose + +Cotabby must understand text and caret geometry inside applications it does not own. macOS +Accessibility is the only cross-application semantic interface available for that job, but its data +is synchronous, cross-process, app-specific, and eventually consistent. This subsystem converts raw +AX state into a bounded, capability-checked FocusSnapshot that the rest of Cotabby can consume. + +## Main Pipeline + +Read these files in order: + +1. [FocusTracker.swift](../../Cotabby/Services/Focus/FocusTracker.swift) +2. [FocusSnapshotResolver.swift](../../Cotabby/Services/Focus/FocusSnapshotResolver.swift) +3. [FocusModels.swift](../../Cotabby/Models/Focus/FocusModels.swift) +4. [AXTextGeometryResolver.swift](../../Cotabby/Services/Focus/AXTextGeometryResolver.swift) +5. [AXHelper.swift](../../Cotabby/Support/Accessibility/AXHelper.swift) + +The high-level flow is: + +~~~text +frontmost or owning application + -> focused AX element + -> candidate editable elements + -> capability and secure-field checks + -> bounded text and selection + -> caret and field geometry + -> stable focus identity and generation + -> FocusSnapshot publication +~~~ + +## Why Polling Is Authoritative + +FocusTracker uses polling as its single source of truth. AXObserver notifications are inconsistent +across native, browser, Electron, and custom editors. Mixing notification and poll streams would +also create ordering ambiguity. Polling provides one eventual-consistency rule: every capture +re-reads the current field, and stale state repairs on a later capture. + +Input and acceptance paths can request refreshNow when they know a fresh read is useful. Those are +still polling-style full captures, not trusted event payloads. FocusTrackingModel exposes the age of +the most recent capture so multiple pipeline stages can avoid paying for identical AX work within a +short freshness window. + +## Adaptive Polling + +Active editing uses the configured base cadence. When repeated captures produce no focus or content +change, FocusPollBackoff stretches the timer interval. Explicit activity resets the backoff. This +keeps editing responsive while avoiding continuous main-thread wakeups and deep AX walks on an idle +machine. + +The timer is re-created only when the effective interval changes. That prevents no-op timer churn on +every keypress. + +## Resolving the Owning Application + +The app that owns the focused AX element is more reliable than NSWorkspace.frontmostApplication. +Accessory utilities such as launchers can own a focused non-activating panel while another +application remains nominally frontmost. Per-app disable policy and surface classification must use +the actual focused process when possible. + +Chromium out-of-process frames complicate ownership because a focused AX node can belong to a +renderer subprocess. The Chromium fallback deliberately associates a hit-tested element with the +visible browser application. + +## Chromium and Electron + +Chromium and Electron surfaces require extra compatibility work: + +- ChromiumAccessibilityEnabler primes web accessibility so renderer text becomes visible. +- A cursor hit-test fallback can recover focused editors that the system-wide focused-element query + misses, including some out-of-process iframe editors. +- The hit-tested element is cached only while it still reports focus and the browser identity + remains valid. +- Browser detection and web-content classification affect candidate choice and geometry policy. +- Debug builds can write a focused Chrome AX tree snapshot for inspection. + +These fallbacks must never mask a real focus change. Every cached element is revalidated, and normal +system focus wins immediately when it becomes available. + +## Candidate Resolution and Capability + +FocusSnapshotResolver searches around the focused element for the most usable editable candidate. +It reduces AX implementation details into domain values: + +- Application and process identity +- Element identity and focus-change sequence +- Preceding and trailing text around the selection +- Selection location and length +- Field and caret rectangles +- Geometry quality +- Window, URL, placeholder, and surface metadata +- Capability and block reason + +FocusCapabilityResolver and related pure policies decide whether the result is supported. Secure +fields, unsupported roles, read-only surfaces, incompatible selections, terminals under default +policy, and explicitly disabled applications must fail safely at the assistance boundary. + +FocusSnapshot intentionally carries only application identity, capability, and optional bounded +context. The obsolete detailed inspection snapshot was removed; developer visibility now travels +through the lightweight FocusPollingEvent plus caret/field geometry and visual-context status. This +keeps normal focus values smaller without removing the debug overlay's polling evidence. + +There is a current acquisition caveat: the resolver constructs a bounded FocusedInputSnapshot before +returning blocked capability for a secure field, and the separate visual-capture gate ignores +capability. Suggestions and insertion remain blocked, but "blocked" does not currently mean that no +AX value or screenshot was acquired. Treat the context/privacy guide as authoritative for that +distinction. + +Cotabby normally blocks its own process. The Context settings pane has one sanctioned live-preview +field identified by a known AX identifier; every other Cotabby field remains excluded. + +## Bounded Text + +Focus snapshots do not carry entire documents. Text on each side of the caret is capped before it +flows through equality checks, Combine publication, stale-result signatures, or prompt construction. +The prompt factory applies a smaller engine-specific window later. + +Bounding at the AX boundary matters because a large editor can otherwise make every focus capture, +comparison, and request signature scale with document size. + +## Focus Identity + +No single AX identifier is perfectly stable. Chromium can recycle or replace AX nodes during normal +editing. Cotabby combines: + +- Process identity +- Element identity where useful +- A monotonic focusChangeSequence +- Selection and text signatures +- Generation/content signatures + +Different consumers use the narrowest identity appropriate to their invariant. Visual context needs +field-scoped uniqueness, while active-session reconciliation intentionally favors process and text +continuity over brittle AX node identity. + +## Geometry Resolution + +AXTextGeometryResolver chooses the best caret geometry available: + +1. Zero-length AXBoundsForRange at the caret +2. AX text-marker range geometry used by some browser engines +3. Bounds of the character before the caret, shifted to its trailing edge +4. Child static-text-run frames with proportional placement +5. Field-frame estimation as a last resort + +The result carries a CaretGeometryQuality: + +- exact: direct caret/range geometry +- derived: geometry inferred from a nearby measured character +- estimated: coarse field-based fallback +- layoutEstimated: repaired later using hidden TextKit layout + +Quality is part of presentation policy. Exact and derived geometry can support inline ghost text. +Estimated or layout-repaired positions normally use a Cotabby-owned mirror card because small +horizontal errors are visually obvious when glyphs are painted directly beside host text. + +## Geometry Cost Controls + +AX calls are synchronous IPC and can block the MainActor while the target process responds. +Compatibility walks therefore use several controls: + +- Gate parameterized attributes on advertised support. +- Validate returned rectangles against an anchor frame. +- Throttle deep descendant searches. +- Throttle static-text-run walks. +- Cache field style and surface metadata within a focus session. +- Invalidate transient caret caches after Cotabby mutates the field. +- Reuse a recent suggestion anchor when AX frames briefly regress after insertion. + +These optimizations are correctness-sensitive. A cache key must include the identity or generation +needed to prevent data from one field leaking into another. + +## App-Specific Safety + +Some applications react badly to AX enumeration. Calendar can dismiss a transient date/time editor +when its tree is walked. CalendarAccessibilityCaptureGuard uses pointer context to suppress only the +fragile interaction window instead of disabling the whole application. + +Per-app disable and global pause are checked before expensive candidate walks. This is both a +performance optimization and a compatibility guarantee: disabled Cotabby should not perturb the +target application's AX tree. + +## Coordinate Systems + +Accessibility, AppKit, Core Graphics, and Vision can describe screen rectangles using different +origins and coordinate conventions. AXHelper and DisplayCoordinateConverter centralize conversion. +Callers should not hand-roll Y-axis flips or mix AX and Cocoa rectangles without an explicit +conversion boundary. + +Every rectangle that reaches AppKit presentation is checked for finite components. A malformed AX +frame should suppress presentation, not crash NSPanel layout. + +## Concurrency and Freshness + +Most AX access is MainActor-isolated. This makes mutation of focus caches and publication ordered, +but it also means expensive synchronous AX calls can affect input responsiveness. The subsystem +therefore emphasizes bounding, throttling, freshness checks, and avoiding duplicate captures. + +Async consumers must assume the snapshot can become stale immediately after they read it. They carry +focus and content signatures through awaits and validate again before applying work. + +## Invariants + +- Polling is the authoritative focus source. +- Cached Chromium fallbacks never outrank a valid system-focused element. +- Secure and unsupported fields fail closed for prediction, presentation, and insertion; early + secure-field acquisition remains a documented privacy limitation. +- Captured text is bounded before publication. +- Deep AX work stops when Cotabby is disabled or interaction safety requires it. +- Geometry always carries a quality classification. +- AX rectangles are converted and validated at explicit boundaries. +- Async results never rely on AX element identity alone. +- Cotabby's own fields remain blocked except for the explicit live-preview field. + +## Failure-Oriented Reading + +- Field is not detected: FocusTracker focus acquisition, Chromium priming/hit test, candidate search. +- Wrong app gets per-app policy: owning-application resolution. +- Secure or read-only field is treated as editable: capability resolver and secure-field detection. +- Ghost appears at field edge: geometry fallback and quality classification. +- Ghost jitters after typing or acceptance: geometry caches, anchor stability, post-insertion + invalidation. +- Idle app consumes CPU: FocusPollBackoff and deep-walk throttles. +- Calendar popover closes: CalendarAccessibilityCaptureGuard and suppression boundary. +- Browser works until iframe focus: Chromium OOPIF hit-test cache and revalidation. + +## Update This Guide When + +Update this document when focus acquisition gains a new source, identity semantics change, a new +geometry branch or quality is added, AX context bounds change architecturally, or an app-specific +compatibility guard changes what Cotabby is allowed to inspect. diff --git a/.internal/architecture/inference-and-prompting.md b/.internal/architecture/inference-and-prompting.md new file mode 100644 index 00000000..8b9c59d3 --- /dev/null +++ b/.internal/architecture/inference-and-prompting.md @@ -0,0 +1,239 @@ +# Inference and Prompting + +## Purpose + +This guide explains how one structured SuggestionRequest becomes a completion through Apple +Intelligence, the in-process llama runtime, or an OpenAI-compatible endpoint. It also describes where +prompt construction, streaming, normalization, cache reuse, credentials, and native resource +ownership belong. + +The key boundary is that engines generate from an immutable request. They do not read live focus or +settings state while decoding. Backend-specific transport and prompting stay behind the shared +SuggestionGenerating contract, while backend-independent output cleanup stays in Support. + +## Main Files + +Start with: + +1. [SuggestionEngineRouter.swift](../../Cotabby/Services/Runtime/SuggestionEngineRouter.swift) +2. [SuggestionRequestFactory.swift](../../Cotabby/Support/Suggestion/SuggestionRequestFactory.swift) +3. [BaseCompletionPromptRenderer.swift](../../Cotabby/Support/Prompting/BaseCompletionPromptRenderer.swift) +4. [FoundationModelPromptRenderer.swift](../../Cotabby/Support/Prompting/FoundationModelPromptRenderer.swift) +5. [SuggestionTextNormalizer.swift](../../Cotabby/Support/Suggestion/SuggestionTextNormalizer.swift) + +Then follow the selected backend: + +- Apple: [FoundationModelSuggestionEngine.swift](../../Cotabby/Services/Runtime/FoundationModelSuggestionEngine.swift) +- In-process llama: [LlamaSuggestionEngine.swift](../../Cotabby/Services/Runtime/LlamaSuggestionEngine.swift), + [LlamaRuntimeManager.swift](../../Cotabby/Services/Runtime/LlamaRuntimeManager.swift), and + [LlamaRuntimeCore.swift](../../Cotabby/Services/Runtime/LlamaRuntimeCore.swift) +- Endpoint: [OpenAICompatibleSuggestionEngine.swift](../../Cotabby/Services/Runtime/OpenAICompatibleSuggestionEngine.swift), + [OpenAICompatibleAPIClient.swift](../../Cotabby/Services/Runtime/OpenAICompatibleAPIClient.swift), + and [OpenAICompatibleEndpointModels.swift](../../Cotabby/Models/Runtime/OpenAICompatibleEndpointModels.swift) + +## Routing + +SuggestionEngineRouter owns backend selection, not model lifecycle or endpoint transport. The +settings snapshot selects one of three paths: + +~~~text +Apple Intelligence + -> FoundationModelSuggestionEngine + +Open Source + -> LlamaSuggestionEngine -> LlamaRuntimeManager -> LlamaRuntimeCore -> CotabbyInference + +OpenAI-compatible endpoint + -> OpenAICompatibleSuggestionEngine -> OpenAICompatibleAPIClient -> configured server +~~~ + +Single-result generation and streaming are both forwarded to the selected backend. Prewarm is sent +only to the selected backend. A context reset is broadcast to all backends because an old continuation +cache must not survive a field or session boundary merely because that backend is temporarily +unselected. + +Apple generation falls back to the in-process llama engine only for unsupported locale or language +conditions. A normal Apple generation failure is surfaced rather than silently changing engines and +making behavior difficult to diagnose. + +## Request Construction + +SuggestionRequestFactory converts a verified focus snapshot and context bundle into an immutable +request. It chooses an engine-appropriate prefix bound, language-aware prediction budget, enabled +context sections, request identifier, and selected-backend prompt payload for explicit developer +debug logging. + +Optional context is sanitized and budgeted before it reaches a renderer. The engine receives both +structured fields and a base-completion prompt so Apple can use its first-class instructions channel +while llama and completion-style endpoints can consume ordinary continuation text. + +The caret prefix is the most important signal. Optional context can be reduced or omitted to preserve +the recent text immediately before the caret. + +## Two Prompt Shapes + +The open-source GGUF models are base completion models, not instruction-tuned chat models. +BaseCompletionPromptRenderer therefore builds a text continuation: + +- A short conditioning preface can describe language, style, or custom rules. +- Optional surface, clipboard, visual, and extended context are bounded by PromptSectionBudget. +- The actual caret prefix appears last so generation continues from the user's text. +- There is no instruction-conversation wrapper that asks the model to obey commands. + +The Apple Foundation Models API supplies a first-class instructions channel. FoundationModelPromptRenderer +keeps instruction-shaped policy separate from the user content instead of pretending the Apple path +is a raw base-model continuation. + +OpenAI-compatible endpoints support completion and chat request modes, but both are derived from the +same Cotabby request and its bounded prompt. Mode changes transport shape; they do not authorize the +endpoint engine to reacquire unbounded editor context. + +## Apple Intelligence + +FoundationModelSuggestionEngine owns Apple framework availability checks, language support, session +creation, prewarm, generation, and streaming. It uses LanguageModelSession.streamResponse for +cumulative partials and retains at most one prepared session for the next compatible request. + +The prepared session is an optimization, not shared conversational memory. A request consumes the +compatible prewarmed session once; field or context resets discard state that could contaminate a new +suggestion. + +Framework or language unavailability is classified distinctly from cancellation and ordinary +generation failure so the router can apply its narrow fallback policy. + +## In-Process llama + +LlamaSuggestionEngine converts the request into the base prompt, invokes the runtime manager, maps +runtime errors into suggestion errors, and passes token-stream partials through normalization before +they reach the coordinator. + +LlamaRuntimeManager is a MainActor ObservableObject. It publishes model discovery, load, warmup, +generation, and failure state for UI consumers. It does not perform tokenization or decoding on the +MainActor. Heavy calls are dispatched away from UI isolation and delegated to LlamaRuntimeCore. + +LlamaRuntimeCore is a lock-protected nonisolated class around mutable native state, not a Swift +actor. Its boundaries are explicit: + +- A lifecycle condition coordinates load, active operations, and bounded shutdown. +- An autocomplete lock serializes prompt-cache and decode state. +- Native pointers and CotabbyInference calls remain private to the core. +- Cancellation is checked through an abort callback while native decoding is running. +- Shutdown prevents new operations, waits for in-flight work, then releases model and context state. + +CotabbyInference is an external SwiftPM wrapper around llama.cpp. It is consumed as a package rather +than vendored in this repository. Pointer ownership, Metal buffers, token buffers, and llama context +correctness stay below the manager boundary. + +## Prompt Cache and Sampling + +The llama core tokenizes the rendered prompt, compares it with the previously evaluated token +sequence, reuses the longest safe common prefix in the KV cache, prefills the remaining prompt, and +then samples new tokens. Token callbacks make partial output available while decode continues. + +Cache reuse is a latency optimization constrained by request continuity. Resetting backend context, +loading another model, or changing to an incompatible prompt invalidates reuse. The cache must never +turn one field's text into hidden context for another field. + +Sampling and stop decisions live inside the native runtime boundary. Output safety still does not: +every backend passes text through the same normalizer before Cotabby presents or inserts it. + +## Runtime Memory Lifecycle + +The local runtime is started only when the selected engine is Open Source. Switching to Apple or an +endpoint stops it so mapped GGUF weights and Metal buffers are released. A model selection change, +download completion, or model-directory refresh can trigger a reload followed by warmup when the +active engine needs it. + +This policy prevents an unused local model from consuming memory while another backend is active. +The manager publishes lifecycle state; AppDelegate decides when engine selection permits the process +to own native resources. + +## OpenAI-Compatible Endpoints + +The endpoint backend supports completion-style and chat-style OpenAI-compatible APIs, including SSE +stream parsing. The configured endpoint may be: + +- Loopback, such as the default Ollama address at http://127.0.0.1:11434/v1 +- A server on the local network +- A public HTTPS service + +Public insecure HTTP is rejected. Configuration models classify endpoint privacy and surface a +warning when text may leave the device. API credentials are stored in Keychain through +OpenAICompatibleCredentialStore rather than in UserDefaults. + +Ollama model discovery and preload are endpoint-specific conveniences. Preload work is coalesced so +repeated settings or warmup events do not launch redundant load requests. Endpoint connection state +is invalidated when URL, model identity, request mode, or credential changes. + +The endpoint path means Cotabby is local-first, not unconditionally offline. Privacy documentation +and UI must distinguish on-device engines from a user-configured remote server. + +## Streaming Contract + +Engines emit cumulative partial text. Each partial is normalized before the coordinator sees it, and +the coordinator accepts only monotonic extensions suitable for replacing the currently visible ghost +tail. UI rendering is coalesced independently of backend token cadence. + +Cancellation is expected when the user types or switches focus. It is not logged or presented as a +runtime failure. A final result is authoritative even after partials were shown; final normalization +or seam checks can replace or suppress the provisional text. + +## Backend-Independent Normalization + +SuggestionTextNormalizer is intentionally outside all engines. It handles: + +- Control and special-token residue +- Reasoning blocks and model scaffolding +- Prompt or prefix echo +- Leading-whitespace reconciliation +- Single-line versus bounded multiline policy +- Text already present after the caret +- Repeated trailing-prefix material +- Empty, unsafe, or non-insertable results + +A new backend must satisfy this same output contract. Backend-specific code should not create a +parallel set of cleanup semantics unless the shared normalizer first receives a genuine domain rule. + +## Diagnostics + +Every request carries a request ID through coordinator, router, engine, runtime, and LLM-I/O logs. +The debug prompt payload comes from the same request factory used for generation, so developer logs +show the actual selected-backend context shape without engines reaching into UI state. It is not +part of the Settings UI or normal user-facing state. + +OSLog metadata is available in normal operation. Full prompts and completions are written to the +debug JSONL sink only when the application is launched with -cotabby-debug. That switch is important +because prompt content can include user text and optional screen or clipboard context. + +## Invariants + +- Routing selects an engine; it does not let engines read live editor state. +- The request is immutable once asynchronous generation starts. +- Base GGUF prompts are continuation-shaped; Apple prompts use its instructions channel. +- The caret prefix is preserved ahead of optional context under budget pressure. +- Heavy llama work never runs on the MainActor. +- Llama native state is serialized under the core's explicit locks and lifecycle condition. +- Context reset prevents cache data from crossing interaction boundaries. +- Switching away from Open Source releases local runtime resources. +- Endpoint secrets live in Keychain and insecure public HTTP is rejected. +- All backends share normalization and streaming semantics. +- Cancellation is a normal lifecycle outcome, not automatically a generation error. + +## Failure-Oriented Reading + +- Wrong engine runs: router selection and settings/profile snapshot. +- Apple unexpectedly falls back: language-support classification in the foundation engine. +- First suggestion is slow: selected-backend prewarm and llama prompt prefill/KV reuse. +- Memory stays high on endpoint mode: AppDelegate runtime stop and manager shutdown. +- Output includes instructions or prompt text: renderer choice and shared normalizer. +- Old field text influences a new field: reset broadcast and llama cache continuity. +- Endpoint never streams: request mode, SSE parsing, and cumulative-partial contract. +- Local endpoint warning looks remote or vice versa: endpoint privacy classification. +- UI freezes during generation: manager-to-core dispatch and synchronous work on MainActor. +- Shutdown crashes in native code: core lifecycle condition and process termination order. + +## Update This Guide When + +Update this document when a backend is added, router fallback policy changes, request fields or +budgets change, prompt shape changes, runtime serialization changes, endpoint privacy rules change, +or streaming and normalization acquire a new cross-backend contract. diff --git a/.internal/architecture/input-and-insertion.md b/.internal/architecture/input-and-insertion.md new file mode 100644 index 00000000..bdca722a --- /dev/null +++ b/.internal/architecture/input-and-insertion.md @@ -0,0 +1,199 @@ +# Input and Insertion + +## Purpose + +This guide explains how Cotabby observes global input, temporarily intercepts only the keys it owns, +and commits accepted text into another application. This is a process-wide boundary: mistakes can +drop user keystrokes, feed Cotabby's synthetic events back into its own state machine, or disturb the +user's clipboard. The design therefore separates observation, consumption, suppression, policy, and +insertion. + +## Main Files + +Read these files in order: + +1. [InputMonitor.swift](../../Cotabby/Services/Input/InputMonitor.swift) +2. [InputSuppressionController.swift](../../Cotabby/Services/Input/InputSuppressionController.swift) +3. [SuggestionInserter.swift](../../Cotabby/Services/Suggestion/SuggestionInserter.swift) +4. [InsertionStrategySelector.swift](../../Cotabby/Support/Suggestion/InsertionStrategySelector.swift) +5. [InsertionSafetyGate.swift](../../Cotabby/Support/Suggestion/InsertionSafetyGate.swift) + +InputMonitor owns event-tap lifetime and converts Core Graphics keyboard events into semantic +CapturedInputEvent values. SuggestionInserter owns the write boundary. The two pure Support policies +decide whether insertion is safe and which mechanism is appropriate without touching global state. + +## Three Event-Tap Responsibilities + +InputMonitor deliberately does not use one permanently consuming event tap. + +~~~text +steady listen-only observer + -> classify ordinary typing, deletion, navigation, and pointer activity + +conditional consuming tap + -> active only while a suggestion or inline-command capture needs interception + -> consume only an event the owning coordinator successfully handles + +dedicated global-toggle tap + -> active only when the shortcut is configured + -> decide the process-wide toggle independently of suggestion visibility +~~~ + +The observer tap is installed at the head of the event chain with listen-only behavior. It can see +events without delaying the focused application or swallowing a key. This is the steady-state path. + +The conditional tap is a default tap because returning nil is the only way to keep an accepted key +from also reaching the host. It is installed only when a visible suggestion claims acceptance keys +or when the emoji picker has an active capture. All unrelated keys pass through unchanged. + +The global-toggle shortcut has a separate consuming tap. Its ownership does not depend on an active +suggestion, and separating it prevents suggestion tap lifetime from changing whether the global +shortcut works. + +## Fail-Open Consumption + +Matching a configured key is not sufficient reason to swallow it. The consuming tap first asks the +current owner to handle the event. It returns nil only when acceptance or capture succeeds. A stale +tap, vanished overlay, changed focus, revoked permission, or rejected session causes the original key +to pass through to the focused application. + +Word/phrase acceptance and full-tail acceptance have independent configurable key and modifier +bindings. Full acceptance takes priority if the bindings overlap. The event is classified using the +current settings at event time so changing a shortcut does not require rebuilding all tap behavior. + +After a final acceptance hides the overlay, the accept tap lingers for a very short deferred teardown. +The callback that consumed the physical key may still be posting synthetic insertion events. Removing +its mach port immediately can prevent the last accepted chunk from reaching the host. The delayed +teardown rechecks whether a suggestion or capture has re-armed the tap before removing it. + +Exhausting a tail also creates a bounded regeneration window in which a rapid follow-up Tab should +not leak into the host and move focus. PostExhaustionAcceptanceState owns the pure arm, queue, +consume, and timeout-generation rules. Repeated presses collapse to one queued accept; the +coordinator owns the timer and interception side effects and returns the key to the host when the +window expires or the session tears down. + +## Inline-Command Capture + +Emoji capture shares the conditional tap but not suggestion acceptance semantics. While a colon +query is active, the emoji controller decides whether navigation, dismissal, or commit keys are +handled. The listen-only observer still routes the event to the coordinator, and the consuming tap +swallows it only after the emoji path reports success. + +This ordering lets Tab remain a configurable suggestion accept key while also committing an emoji +selection during capture. InputMonitor knows which subsystem currently owns interception; it does not +implement emoji query or suggestion-session rules itself. + +## Suppressing Cotabby's Own Events + +[InputSuppressionController.swift](../../Cotabby/Services/Input/InputSuppressionController.swift) +prevents inserted text from being mistaken for fresh user typing. SuggestionInserter marks every +synthetic Core Graphics event with a Cotabby-specific source value and registers an expected event +burst before posting it. + +The marker is the strongest identity check. The bounded suppression countdown is a compatibility +fallback for event transformations that do not preserve every property. It accumulates across rapid +acceptances because global event delivery is asynchronous; overwriting the count could expose the +tail of an earlier insertion as apparent user input. + +Suppression is intentionally narrow and expires. It must not hide real typing after an insertion or +become a general pause in focus reconciliation. + +## Pre-Insertion Safety + +The coordinator validates the active session, overlay, focus, and current text before asking for a +write. InsertionSafetyGate provides the pure final checks for the proposed chunk. Invalid or unsafe +text is rejected before an event is posted. + +This boundary does not assume that generation success authorizes insertion. The user may have moved +the caret, selected text, switched applications, or edited the field after the suggestion became +visible. + +## Default Unicode Keystroke Path + +For the common short, single-line continuation, SuggestionInserter posts a Unicode key-down/key-up +pair through Core Graphics. This path is app-agnostic and does not touch the clipboard. It also avoids +relying on an application's support for direct Accessibility value mutation. + +The events use a placeholder virtual key because the committed payload is carried as Unicode text. +That detail is why the synthetic marker matters: a user could bind acceptance to the same key code, +and Cotabby must never consume its own insertion as another acceptance. + +## IME-Safe and Optional Paste Paths + +Synthetic Unicode events can be intercepted as composing input while an input method editor is +active. In that state, SuggestionInserter commits through a clipboard paste so accepted Japanese, +Chinese, or other composed text lands as final text rather than being fed back into composition. + +There is also a default-off paste strategy for long or multiline suggestions. When enabled, +InsertionStrategySelector chooses paste for multiline text or chunks at least 80 characters long. +Short ordinary completions remain on the clipboard-free Unicode path. + +The paste path: + +1. Snapshots every representation of every item on the general pasteboard. +2. Replaces the pasteboard temporarily with the accepted plain text. +3. Tries the target application's Accessibility Edit > Paste menu action first. +4. Falls back to a synthetic Command-V when the menu item cannot be pressed. +5. Restores the saved clipboard after the host has had time to service the paste. + +Pressing the actual Paste menu item is important for applications such as Chrome, where a synthetic +Command-V posted from the global tap callback can fail while a physical accept key is still down. +The menu item is cached per process only after its AXPress result is validated. + +Clipboard restoration is change-count aware. If the user or another application changes the +clipboard before restoration, Cotabby leaves the newer contents alone. Overlapping paste insertions +reuse the one saved user clipboard instead of accidentally snapshotting Cotabby's previous completion +and restoring that to the user. + +## Replacement Writes + +A normal suggestion inserts a continuation at the caret. A spelling-correction or inline-command +session can instead replace a known literal run. The inserter posts the required deletion burst and +then the replacement text under the same suppression boundary. + +Replacement length is derived from the validated session, not from a new untrusted scan at write +time. The coordinator still reconciles the host's later AX publication because posting events proves +only that Cotabby requested the mutation, not that every application applied it exactly as expected. + +## Permissions and Recovery + +Input Monitoring permission governs global event observation. Accessibility is also required for +focus validation and for the preferred paste-menu action. If taps are disabled by timeout or user +input, InputMonitor re-enables them where safe. If permission is revoked, the consuming tap is torn +down and later recreated only after the permission state allows it. + +No recovery path may leave a dead consuming tap installed. A tap that cannot confidently handle an +event must pass it through. + +## Invariants + +- Steady-state observation is listen-only. +- A key is consumed only after the owning feature reports success. +- Post-exhaustion Tab ownership is bounded, generation-keyed, and can queue at most one accept. +- The consuming suggestion/capture tap exists only while interception is needed. +- Global toggle ownership is independent of suggestion visibility. +- Every synthetic event is tagged and covered by bounded self-event suppression. +- Short ordinary completions use the clipboard-free Unicode path. +- IME-active insertion uses a commit mechanism that bypasses composition. +- Paste restores all saved pasteboard representations unless newer clipboard activity wins. +- Insertion never trusts a generation result without revalidating the live session. +- Host publication after insertion is reconciled rather than assumed. + +## Failure-Oriented Reading + +- Acceptance key is swallowed with no visible suggestion: fail-open authorization in InputMonitor. +- Accepted text triggers another prediction as typing: synthetic marker and suppression accounting. +- Final accepted chunk disappears: deferred accept-tap teardown and event posting. +- Rapid Tab moves host focus after the visible tail ends: post-exhaustion acceptance window and + backstop generation. +- Tab commits the suggestion instead of emoji: capture ownership and accept-tap resolution order. +- IME acceptance regenerates or enters composition: active-input-source detection and paste path. +- Clipboard contains a completion afterward: snapshot, overlap, change-count, and restore logic. +- Chrome ignores paste: AX Paste menu lookup before Command-V fallback. +- Correction removes the wrong characters: replacement session validation and deletion count. + +## Update This Guide When + +Update this document when an event tap gains a new ownership reason, acceptance bindings change +semantics, suppression identity changes, a new insertion strategy is added, clipboard restoration +policy changes, or another inline feature begins consuming global input. diff --git a/.internal/architecture/lifecycle-and-composition.md b/.internal/architecture/lifecycle-and-composition.md new file mode 100644 index 00000000..4183485e --- /dev/null +++ b/.internal/architecture/lifecycle-and-composition.md @@ -0,0 +1,195 @@ +# Lifecycle and Composition + +## Purpose + +This guide explains who creates Cotabby's long-lived objects, who starts and stops them, and why +views do not construct services. It is the first deep dive to read after the root architecture map. + +Cotabby integrates with process-wide macOS facilities: Accessibility, global event taps, screen +capture, Sparkle, and an in-process native model runtime. Accidentally constructing two copies of +one of those services can produce duplicate observers, competing event taps, runtime reload races, +or two pieces of UI that disagree about current settings. The composition root exists to prevent +that class of bug. + +## Primary Owners + +The lifecycle is divided among three files: + +1. [CotabbyApp.swift](../../Cotabby/App/Core/CotabbyApp.swift) declares the SwiftUI scene graph and + bridges the AppKit delegate into the SwiftUI application. +2. [AppDelegate.swift](../../Cotabby/App/Core/AppDelegate.swift) owns process lifecycle callbacks, + starts and stops services, and wires reactions that are specifically tied to application + lifecycle. +3. [CotabbyAppEnvironment.swift](../../Cotabby/App/Core/CotabbyAppEnvironment.swift) constructs the + dependency graph once and retains app-scoped subscriptions between the objects it created. + +The split is intentional. Construction answers "which concrete objects satisfy the app's +dependencies?" Lifecycle answers "when may those objects begin side effects?" SwiftUI answers +"which state should be presented?" Keeping those questions separate makes ownership visible. + +## Process Startup + +CotabbyApp is the process entry point. It creates AppDelegate through NSApplicationDelegateAdaptor +and declares the MenuBarExtra scene. SwiftUI owns insertion and removal of the status item, while +SuggestionSettingsModel remains the durable source of truth for whether the icon is visible. + +AppDelegate initialization performs synchronous graph construction: + +~~~text +CotabbyApp + -> AppDelegate.init + -> CotabbyLogger.bootstrap + -> CotabbyAppEnvironment.init + -> expose shared environment objects to SwiftUI + -> install cross-subsystem closures and subscriptions +~~~ + +Constructing the graph does not start global event processing. AppDelegate waits for +applicationDidFinishLaunching before it: + +1. Applies the one-time Open at Login default. +2. Starts the llama runtime only when the selected engine requires it. +3. Starts focus polling. +4. Starts global input monitoring. +5. Starts the update manager. +6. Starts the suggestion coordinator. +7. Starts the inline-command coordinator. +8. Presents onboarding or permission reminders when needed. +9. Restores a visible Settings surface when the menu bar icon is hidden and launch policy requires + recovery. + +The XCTest host is a special case. App-hosted unit tests launch the real application binary before +loading the test bundle. AppDelegate detects that environment and skips production service startup +so tests do not install global taps, begin polling, launch Sparkle, or load a model as a side effect. + +## The Environment Graph + +CotabbyAppEnvironment is MainActor-isolated because most graph members publish UI state, call +AppKit, or access Accessibility. It constructs shared services in dependency order: + +~~~text +settings and permission state + -> input/focus services + -> insertion and presentation services + -> context services + -> generation engines and router + -> suggestion coordinator + -> emoji and macro controllers + -> settings/onboarding coordinators +~~~ + +Important shared instances include: + +- PermissionManager and PermissionGuidanceController +- SuggestionSettingsModel +- LlamaRuntimeManager wrapped by RuntimeBootstrapModel +- ModelDownloadManager +- FocusTrackingModel and InputMonitor +- InputSuppressionController and SuggestionInserter +- OverlayController and ActivationIndicatorController +- ClipboardContextProvider and VisualContextCoordinator +- SuggestionEngineRouter with all generation backends +- SuggestionCoordinator, SuggestionInteractionState, and SuggestionWorkController +- EmojiPickerController, MacroController, and InlineCommandCoordinator +- SettingsCoordinator and WelcomeCoordinator + +The environment passes narrow protocol-shaped collaborators into SuggestionCoordinator. This keeps +the coordinator testable without turning every concrete service into a global singleton. + +## Where Subscriptions Live + +Both the environment and AppDelegate retain Combine subscriptions. That is not accidental +duplication; the dividing line is ownership. + +CotabbyAppEnvironment retains relationships among objects it constructed: + +- Settings changes update the focus poll interval. +- Binding or clearing the global-toggle shortcut installs or removes its dedicated event tap. +- Power-source and profile changes select the appropriate engine and model. +- Selecting the endpoint engine triggers model discovery. +- Endpoint identity or credential changes invalidate connection state. + +AppDelegate retains process-lifecycle reactions: + +- Permission changes refresh input monitoring. +- Engine changes start or release the in-process llama runtime. +- Focus changes update activation and debug overlays. +- Settings changes hide the activation indicator when Cotabby becomes disabled or paused. +- Model-directory changes refresh available runtime models. +- Visual-context state is mirrored into the debug overlay. + +CotabbyAppEnvironment itself must therefore be retained for the whole process. If it were a +temporary local variable in AppDelegate.init, its cancellables would deallocate and settings-driven +behavior would silently stop. + +## Runtime Selection and Memory Ownership + +The in-process llama runtime is warmed only when the selected engine is Open Source. Selecting Apple +Intelligence or an OpenAI-compatible endpoint stops the local runtime so mapped model weights and +Metal buffers are released. Model downloads or manual file changes trigger a rescan, followed by a +warmup only if the current engine needs llama. + +Power-based profiles are applied in the environment. A profile can select Apple Intelligence, +an installed GGUF model, or an endpoint model. Applying a profile is idempotent so repeated +publisher emissions do not cause unnecessary model reloads. + +## Shutdown + +applicationWillTerminate reverses process-wide side effects: + +1. Hide activation and debug overlays. +2. Stop suggestion and inline-command coordination. +3. Stop input monitoring. +4. Stop focus polling. +5. Synchronously release the native llama runtime with a bounded wait. + +The synchronous native shutdown is a correctness requirement. Exiting while llama/Metal resources +remain alive can collide with C++ static destruction. At the same time, shutdown cannot defer +application termination indefinitely because macOS permission flows rely on a prompt quit and +relaunch to observe a new TCC grant. + +## Settings and Presentation Ownership + +SuggestionSettingsStore persists durable non-secret values in UserDefaults. +SuggestionSettingsModel keeps individually `@Published` properties for SwiftUI and source +compatibility. Its `domainSettings` projection builds a `SuggestionSettingsData` value grouped into +general, engine, completion, context, correction, presentation, inline-feature, and shortcut +domains. The projection does not duplicate storage or change publication timing. + +SuggestionSettingsSnapshot is derived from those domains and remains the immutable behavior input +for the suggestion pipeline. SuggestionSettingsStore deliberately maps the domain values back to +the existing flat UserDefaults keys so the refactor does not migrate or invalidate persisted user +preferences. OpenAI-compatible credentials are kept in Keychain rather than UserDefaults. + +Views observe shared models or receive callbacks from coordinators. They do not instantiate focus, +input, runtime, permission, download, or update services. AppKit window controllers are likewise +constructed at app scope when their lifetime must outlive a SwiftUI view redraw. + +## Invariants + +- There is one production instance of each process-wide observer, tap, coordinator, and runtime. +- Construction does not imply startup; AppDelegate controls when side effects begin. +- SwiftUI view reconstruction must not reconstruct services. +- The environment remains retained as long as its subscriptions are needed. +- Heavy model, OCR, download, and file-copy work must not block the MainActor. +- Domain settings are projections over one durable source, not a second mutable settings graph. +- Switching away from llama releases native runtime resources. +- Production services do not start inside the XCTest host. +- Shutdown stops new work before releasing native state. + +## Failure-Oriented Reading + +- Duplicate taps or observers: start in CotabbyAppEnvironment and AppDelegate initialization. +- A setting changes but behavior does not: inspect the environment's retained cancellables. +- Runtime remains loaded on another engine: inspect AppDelegate engine switching. +- Hidden menu bar icon makes the app unreachable: inspect MenuBarRecoveryPolicy and reopen handling. +- Permission relaunch loops or termination crashes: inspect AppDelegate termination and permission + guidance. +- A view appears to own a service: trace construction back to CotabbyAppEnvironment before changing + view lifetime. + +## Update This Guide When + +Update this document when a new app-lifetime service is added, startup or shutdown order changes, +ownership moves between AppDelegate and CotabbyAppEnvironment, a new engine affects runtime memory +policy, or a SwiftUI scene begins owning behavior instead of presentation. diff --git a/.internal/architecture/presentation-and-sibling-features.md b/.internal/architecture/presentation-and-sibling-features.md new file mode 100644 index 00000000..3776240b --- /dev/null +++ b/.internal/architecture/presentation-and-sibling-features.md @@ -0,0 +1,241 @@ +# Presentation and Sibling Features + +## Purpose + +This guide covers Cotabby-owned UI that appears around another application's editor and the features +that share global input infrastructure with suggestions. The presentation layer is a deliberate mix +of SwiftUI and AppKit: SwiftUI describes content, while AppKit owns non-activating panels, window +levels, focus behavior, and geometry that ordinary SwiftUI scenes cannot express reliably. + +## Presentation Ownership + +The main suggestion presentation path is: + +~~~text +SuggestionCoordinator + -> SuggestionOverlayPresenter + -> OverlayController + -> non-activating NSPanel + -> SwiftUI ghost or mirror content +~~~ + +[SuggestionOverlayPresenter.swift](../../Cotabby/Services/Suggestion/SuggestionOverlayPresenter.swift) +adapts coordinator intent into show, move, update, or hide actions and returns diagnostic reasons. +It owns the small state-diff rules, not AppKit window construction. + +[OverlayController.swift](../../Cotabby/Services/Presentation/OverlayController.swift) owns the reusable +borderless panel, SwiftUI hosting views, render-mode selection, layout, font and color resolution, +fade behavior, and visible OverlayState. It is the only owner allowed to treat the suggestion as an +AppKit window. + +The panel is non-activating, ignores mouse events, joins all Spaces, can appear over full-screen apps, +and does not participate in the window cycle. Showing a suggestion must never take keyboard focus +from the editor Cotabby is assisting. + +## Inline and Mirror Modes + +[CompletionRenderModePolicy.swift](../../Cotabby/Support/Presentation/CompletionRenderModePolicy.swift) chooses +how to render from user preference and geometry quality. + +- Inline mode paints ghost text immediately beside a trusted caret. Exact and derived geometry are + precise enough for this path under automatic policy. +- Mirror mode presents a Cotabby-owned card near the text line when geometry is estimated, + layout-estimated, intentionally forced by user preference, or the caret is mid-line where free + ghost glyphs could overlap existing trailing text. + +The mirror card is not pretending to be host text. It trades visual continuity for correctness when +the exact glyph insertion point is unknown. MirrorOverlayLayout uses the reason for promotion to +choose an anchor and keep the card inside the visible screen. + +Per-application render-mode overrides are represented as future policy capacity but are not wired in +the current product. Documentation should not claim they ship until the focused bundle identifier is +actually passed to the policy. + +## Field-Matched Rendering + +Focus resolution can carry a ResolvedFieldStyle with font family, weight, size, and foreground color. +Inline rendering uses that style when available and derives size primarily from caret height. It +falls back to a system font when the host exposes no usable style. + +Presentation also supports: + +- User-selected ghost color and opacity +- A bounded size multiplier +- Configurable acceptance-key hint +- Optional fade-in duration, respecting Reduce Motion +- Green correction styling for replace-the-word suggestions +- Right-to-left placement +- Bounded multiline layout and wrapping inside the field or screen + +These are presentation settings. They do not enter SuggestionRequest or change generated text. + +## Stable Partial and Acceptance Updates + +Streaming can update many times per second. The coordinator coalesces partials and the presenter +skips identical text/geometry. OverlayController reuses separate typed NSHostingView instances for +inline and mirror roots so token extensions do not allocate a new window or hosting hierarchy. + +For a single-line left-to-right inline suggestion, word acceptance and exact type-through can advance +the remaining tail by the measured width of the committed prefix. That keeps the ghost visually +stationary relative to the user's typing instead of waiting for a noisy AX caret refresh. + +[SuggestionOverlayStabilityGate.swift](../../Cotabby/Support/Presentation/SuggestionOverlayStabilityGate.swift) +decides whether later reconciliation should hold the existing anchor or re-anchor to fresh geometry. +It tolerates the known pre-insertion AX frame briefly and corrects meaningful drift without allowing +small polling noise to make the tail jitter. + +Mirror and multiline cases fall back to fresh layout when a simple horizontal advance is not valid. + +## Activation and Debug Overlays + +[ActivationIndicatorController.swift](../../Cotabby/Services/Presentation/ActivationIndicatorController.swift) +owns the optional caret or field-edge availability indicator. It follows supported focus and is +hidden when Cotabby is disabled, paused, missing required permissions, or not eligible. + +[FocusDebugOverlayController.swift](../../Cotabby/Services/Presentation/FocusDebugOverlayController.swift) +shows lightweight polling sequence/capability, caret and field geometry, build identity, and +visual-context status for development. The old detailed FocusInspectionSnapshot and published +suggestion-diagnostics surface are not part of this path. It is gated by -cotabby-debug and must not +become normal user-facing settings state. + +These controllers use their own non-activating panels because their visibility and layout lifetimes +are independent from suggestion text. + +## Inline Command Ownership + +[InlineCommandCoordinator.swift](../../Cotabby/App/Coordinators/InlineCommandCoordinator.swift) fans +captured input to the emoji and macro controllers. InputMonitor exposes one capture-decision slot and +one capture-interception flag; the coordinator prevents the two features from competing for those +shared resources. + +Emoji and macro sigils are disjoint, so at most one capture should be active. The coordinator routes +the consuming-tap decision to the current owner and reports whether either feature was involved so +SuggestionCoordinator can stand down for that event. + +Both features: + +- Check live settings before triggering. +- Require a supported, non-secure focus context. +- Pin capture to a focus-change sequence. +- Cancel on incompatible navigation, focus change, timeout, or dismissal. +- Replace the literal typed run through SuggestionInserter. +- Use pure Support state machines for keystroke-to-action rules. + +## Emoji Picker + +[EmojiPickerController.swift](../../Cotabby/App/Coordinators/EmojiPickerController.swift) owns one +colon-query capture. [EmojiTriggerStateMachine.swift](../../Cotabby/Support/Emoji/EmojiTriggerStateMachine.swift) +decides when a colon may open a query and how typing, deletion, navigation, closing colon, acceptance, +and dismissal change the capture. + +[EmojiCatalog.swift](../../Cotabby/Support/Emoji/EmojiCatalog.swift) lazily loads the bundled resource. +[EmojiMatcher.swift](../../Cotabby/Support/Emoji/EmojiMatcher.swift) ranks query matches using names, +keywords, synonyms, popularity, recency, and usage without owning UI. + +[EmojiPickerPanelController.swift](../../Cotabby/Services/Presentation/EmojiPickerPanelController.swift) owns the +non-activating picker panel and SwiftUI content. Arrow keys move selection, the configured word-accept +key commits a match, and a supported closing-colon form can commit the best match. The controller then +replaces the literal colon query and records recency/frequency in EmojiUsageStore. + +Catalog and matcher initialization are lazy so users who never type an emoji query do not pay the +resource and index cost at launch. + +## Macros + +[MacroController.swift](../../Cotabby/App/Coordinators/MacroController.swift) owns slash-query capture +and one-row preview presentation. [MacroTriggerStateMachine.swift](../../Cotabby/Support/Macros/MacroTriggerStateMachine.swift) +defines capture semantics, including guards that prevent ordinary URL fragments, fractions, and +mid-word slashes from triggering. + +[MacroEngine.swift](../../Cotabby/Support/Macros/MacroEngine.swift) tries deterministic evaluator +families in priority order: + +- Date and time expressions +- Random values +- Unit conversion +- Currency conversion +- Arithmetic + +Clock, calendar, locale, and random source are injectable, so the engine remains deterministic under +test. A successful result carries preview text and replacement text; accepting it replaces the typed +slash query rather than invoking any language model. + +## Settings + +[SettingsCoordinator.swift](../../Cotabby/App/Coordinators/SettingsCoordinator.swift) owns the single +AppKit settings window and hosts [SettingsContainerView.swift](../../Cotabby/UI/Settings/SettingsContainerView.swift). +The window survives SwiftUI view reconstruction, remembers placement through AppKit, and routes +permission actions to PermissionGuidanceController. + +Settings panes under Cotabby/UI/Settings/Panes remain presentation-focused. They bind to shared +models and call narrow mutation methods. Search indexing, attention callouts, hardware recommendations, +and validation rules belong in Support or Models so the view hierarchy does not become the source of +product behavior. + +SuggestionSettingsModel retains individually published properties for existing SwiftUI bindings. +Its domainSettings projection groups the same values by the subsystem that owns each decision, and +SuggestionSettingsSnapshot supplies the smaller immutable behavior surface used by the pipeline. +SuggestionSettingsStore remains the sole UserDefaults owner and keeps persisted keys flat for +compatibility. + +ContextLivePreviewField is the one Cotabby-owned editable field permitted through focus capability +for live preview. The exception is identified explicitly; general Cotabby windows remain blocked from +autocomplete to prevent the app from assisting itself. + +## Onboarding + +[WelcomeCoordinator.swift](../../Cotabby/App/Coordinators/WelcomeCoordinator.swift) owns the onboarding +and required-permission reminder windows. It restores incomplete progress, presents the correct step, +resizes the AppKit window to SwiftUI content, and keeps permission guidance attached to the active +surface. + +Onboarding templates recommend a settings configuration; they do not create a separate runtime or +coordinator graph. Applying a template mutates the shared settings model so the same lifecycle +subscriptions and validation rules run as they would for manual changes. + +Input Monitoring can require a quit/relaunch before the new TCC state is effective. WelcomeCoordinator +persists progress and returns the user to the permission step rather than treating that process exit +as abandoned onboarding. + +## Menu Bar and Reachability + +[CotabbyApp.swift](../../Cotabby/App/Core/CotabbyApp.swift) declares the SwiftUI MenuBarExtra. Status +item visibility is persisted in shared settings. Because hiding the status item can make an agent app +hard to reach, AppDelegate and MenuBarRecoveryPolicy restore a settings surface under defined launch +or reopen conditions. + +This is a lifecycle rule, not merely a view preference: any new way to hide the menu icon must retain +a reliable path back into Settings. + +## Invariants + +- Cotabby-owned panels never steal focus from the assisted editor. +- SuggestionOverlayPresenter decides presentation actions; OverlayController owns AppKit mechanics. +- Automatic mode uses inline only when geometry and editing position make it safe. +- Streaming updates reuse presentation objects and avoid redundant renders. +- Partial acceptance updates the remaining tail synchronously. +- Debug presentation remains behind the explicit debug launch gate. +- Emoji and macro capture never compete for the one consuming-tap slot. +- Inline commands require a supported non-secure field and stay pinned to one focus sequence. +- Pure query and evaluation rules remain outside AppKit controllers. +- Settings and onboarding observe the app-scoped graph rather than constructing services. +- Hiding the menu bar item never permanently removes access to settings. + +## Failure-Oriented Reading + +- Ghost steals application focus: OverlayPanel configuration and show calls. +- Ghost overlaps host text mid-line: completion render-mode policy and geometry flags. +- Card appears despite exact caret: user mirror preference and quality passed into policy. +- Tail jitters after acceptance: advanceInline and SuggestionOverlayStabilityGate. +- Wrong font or color: resolved field style capture and overlay fallback. +- Emoji and suggestion both accept Tab: InlineCommandCoordinator capture ownership. +- Colon in a URL opens the picker: EmojiTriggerStateMachine boundary rules. +- Slash in a fraction opens macros: MacroTriggerStateMachine boundary rules. +- Settings opens duplicate windows: SettingsCoordinator lifetime. +- Onboarding restarts from the wrong step: WelcomeCoordinator persisted progress. +- Hidden icon leaves no entry point: MenuBarRecoveryPolicy and application reopen handling. + +## Update This Guide When + +Update this document when a render mode, panel, inline feature, settings surface, onboarding step, or +menu-bar reachability rule is added or changes ownership. diff --git a/.internal/architecture/suggestion-pipeline.md b/.internal/architecture/suggestion-pipeline.md new file mode 100644 index 00000000..e233538c --- /dev/null +++ b/.internal/architecture/suggestion-pipeline.md @@ -0,0 +1,265 @@ +# Suggestion Pipeline + +## Purpose + +This guide follows one autocomplete attempt from an input event to visible or inserted text. The +pipeline is a state machine operating over eventually consistent Accessibility data. Reliability +comes from explicit work identity, cancellation, focus signatures, session reconciliation, and +safe failure behavior rather than from assuming that events arrive in a convenient order. + +## Coordinator Reading Order + +Read the coordinator in this order: + +1. [SuggestionCoordinator.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator.swift) +2. [SuggestionCoordinator+Lifecycle.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator+Lifecycle.swift) +3. [SuggestionCoordinator+Input.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator+Input.swift) +4. [SuggestionCoordinator+Prediction.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator+Prediction.swift) +5. [SuggestionCoordinator+Acceptance.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator+Acceptance.swift) + +The base file declares dependencies and mutable orchestration state. The extensions group lifecycle, +input, prediction, and acceptance behavior without turning those concerns into separate owners. + +## End-to-End Flow + +~~~text +focus snapshot or input event + -> availability and session reconciliation + -> debounce and work identity + -> refresh AX state if stale + -> native correction decision + -> bounded request construction + -> selected engine generation + -> streamed partial presentation + -> authoritative final normalization and guards + -> active suggestion session + -> overlay presentation + -> type-through, dismissal, partial acceptance, or full acceptance + -> host publish reconciliation and next prediction +~~~ + +## Starting and Stopping + +start installs callbacks on the focus provider, input monitor, permissions, settings, overlay, and +visual-context coordinator. It also snapshots current settings and begins reacting to the current +focus. stop removes or neutralizes those callbacks, cancels all prediction work, clears active +sessions, hides presentation, and resets backend-local generation context. + +The coordinator is MainActor-isolated. That makes transitions involving coordinator state, AppKit +presentation, and Accessibility snapshots sequential from the coordinator's perspective. Only +state that UI actually observes remains `@Published`; internal debug/session values are plain +MainActor properties. Heavy backend work is moved off the actor by the backend implementations. + +## Focus Changes and Prewarm + +A supported focused field creates or refreshes a field-scoped interaction context. Focus changes: + +- Cancel obsolete generation work. +- Reset active suggestion and backend continuation state when continuity is broken. +- Start visual-context capture for the new field when enabled and permitted. +- Build a request-shaped warmup payload. +- Prewarm only the selected backend. + +Prewarm is opportunistic. Failure to warm never becomes a user-facing prediction failure; it only +means the first real request pays the cold setup cost. + +## Input Handling + +Inline commands receive first look at captured input. If the emoji or macro feature owns the current +keystroke, normal suggestion behavior stands down and any conflicting ghost text is hidden. + +For ordinary input, the coordinator distinguishes: + +- Direct text mutation +- Deletion +- Navigation or selection movement +- Escape/dismissal +- Word acceptance +- Full acceptance +- Synthetic events Cotabby generated itself + +Direct typing can advance a visible session without regenerating when the typed characters exactly +match the next suggestion tail. Divergent typing invalidates the session and schedules new work. +Navigation, selection, focus changes, or incompatible trailing-text changes clear stale sessions. + +## Work Identity and Cancellation + +[SuggestionWorkController.swift](../../Cotabby/Services/Suggestion/SuggestionWorkController.swift) +owns the debounce task, generation task, and a monotonically increasing work ID. + +Replacing work: + +1. Cancels the previous debounce and generation tasks. +2. Advances the work ID. +3. Starts a new debounce operation. +4. Allows generation to apply results only while that ID remains current. + +Cancellation alone is insufficient because native or system APIs can finish after Swift cancellation +was requested. Result application therefore also validates work ID, request generation, focus +identity, content signature, settings continuity, and current field state. + +## Debounce and AX Publish Timing + +A global key event can arrive before the host application publishes its new value through +Accessibility. Reading immediately would build a request from pre-keystroke text. Cotabby combines +debounce with explicit focus refreshes and short host-publish polling when it knows the host is +catching up. + +The pipeline has a ceiling: if a host never publishes a detectable change, it eventually proceeds +through normal downstream guards instead of waiting forever. Freshness helpers avoid paying +duplicate synchronous AX walks when another pipeline stage just captured the same state. + +## Eligibility + +[SuggestionAvailabilityEvaluator.swift](../../Cotabby/Support/Suggestion/SuggestionAvailabilityEvaluator.swift) +contains pure gating rules. A prediction may be withheld because of: + +- Missing required permissions +- Global disable or temporary pause +- Per-application disable +- Unsupported or secure focus capability +- Selection or incompatible editing state +- Terminal policy +- Insufficient text signal +- Runtime or engine availability + +Gating is repeated at meaningful async boundaries. Passing eligibility before debounce does not +authorize applying a result after the user switches fields, changes settings, or selects text. + +## Corrections Before Model Generation + +Spelling correction is a native fast path evaluated before model generation. TypoGate determines +whether the current editing shape is eligible. CurrentWordSpellChecker uses NSSpellChecker, while +SymSpellCorrector supplies frequency-ranked multilingual candidates after its index is available. + +Depending on settings and the input event, the coordinator can: + +- Suppress model completion while the user is still building a likely typo. +- Present a correction as a green replace-the-word suggestion. +- Automatically replace a completed typo after Space. +- Continue to ordinary model generation when no correction applies. + +A correction session has different acceptance semantics from a continuation. It commits as one +atomic replacement rather than exposing partial word acceptance. + +## Request Construction + +[SuggestionRequestFactory.swift](../../Cotabby/Support/Suggestion/SuggestionRequestFactory.swift) builds an +immutable SuggestionRequest and the selected backend's developer-debug prompt payload. It: + +- Bounds the prefix according to the selected engine. +- Selects a language-aware prediction budget. +- Adds enabled extended, clipboard, visual, and surface context. +- Sanitizes and budgets optional context. +- Builds the base-model prompt used by llama and completion-style endpoints. +- Preserves structured request fields for the Apple prompt renderer. +- Assigns a request ID used across structured logs. + +The coordinator decides when to request. The factory decides what the request contains. Engines do +not reach back into live Accessibility state while generation is running. + +## Streaming and Final Results + +SuggestionGenerating exposes both a single-result method and a streaming method. Streaming engines +send cumulative, already-normalized partials. +[SuggestionStreamingState.swift](../../Cotabby/Support/Suggestion/SuggestionStreamingState.swift) +owns three pure bookkeeping rules: the newest pending partial wins, only one runloop drain is +scheduled at a time, and rendered text grows monotonically through StreamedGhostTextPolicy. The +coordinator still owns DispatchQueue scheduling, freshness checks, session creation, and overlay +side effects. + +A streamed partial can become a real active session, allowing acceptance before decoding finishes. +The final result remains authoritative. It may replace the partial, suppress it, or clear it if final +confidence and seam guards reject the completion. + +Streaming presentation is controlled by settings. Backend streaming support does not imply that +partials must always be painted. + +## Normalization and Display Guards + +SuggestionTextNormalizer applies backend-independent cleanup: + +- Removes control tokens and reasoning blocks. +- Strips echoed prompt scaffolding. +- Applies single-line or bounded multi-line policy. +- Rejects duplication of text already after the caret. +- Removes repeated prefix echoes. +- Reconciles leading whitespace. +- Rejects unsafe or empty insertions. + +The coordinator then applies display-time checks such as CompletionSeamGuard. That guard suppresses +junk punctuation runs and likely mid-word misspelling splices. Streaming uses the cheap pure part of +the guard; the authoritative final can run the full spelling-dependent verdict. + +## Active Sessions + +[SuggestionInteractionState.swift](../../Cotabby/Services/Suggestion/SuggestionInteractionState.swift) +owns: + +- The materialized focus context buffer +- The active suggestion session +- Consumed character count +- The sentinel indicating that Cotabby inserted text but AX has not published it yet + +[SuggestionSessionReconciler.swift](../../Cotabby/Support/Suggestion/SuggestionSessionReconciler.swift) +contains the pure rules for comparing a session with live editor state. It tolerates narrowly scoped +post-insertion AX lag while rejecting focus changes, altered trailing text, selection, undo, or +divergent typing. + +## Acceptance + +The word-accept key follows the configured word or phrase granularity. The full-accept key commits +the entire remaining tail. Acceptance validates: + +- Cotabby is still enabled. +- Input Monitoring remains granted. +- The focused field remains supported. +- A live session exists. +- The visible overlay matches the remaining session text. +- Current AX text still reconciles with the session anchor. + +Successful insertion advances or exhausts the session. Partial acceptance updates the visible tail +synchronously so a rapid second press cannot observe an overlay/session mismatch. + +After final acceptance, Cotabby starts speculative generation against the text the host is expected +to publish. A parallel publish check validates that guess. If the host publishes different content, +ordinary newer work supersedes the speculation. + +There is also a short gap between exhausting the visible tail and presenting a regenerated +continuation. [PostExhaustionAcceptanceState.swift](../../Cotabby/Support/Suggestion/PostExhaustionAcceptanceState.swift) +keeps Tab owned only during that bounded window, collapses repeated rapid presses into at most one +queued accept, and keys the timeout to the current arm generation. The coordinator owns the event-tap +and timer effects; the value owns only the transition rules. A timeout or teardown returns Tab to the +host, while a fresh continuation atomically consumes the queued accept. + +## Invariants + +- Only the current work ID may apply asynchronous results. +- Focus and content signatures are revalidated after awaits. +- An overlay is acceptable only when it matches the active session tail. +- Streamed partials are provisional; the final result is authoritative. +- A cancellation is not automatically a runtime failure. +- Stream drain and post-exhaustion acceptance rules live in pure state values; their timers, + scheduling, input interception, and presentation effects remain coordinator-owned. +- Native correction runs before model generation. +- AX lag tolerance is narrow and tied to a known Cotabby insertion. +- Pure policy stays in Support; mutable orchestration stays in the coordinator and service owners. + +## Failure-Oriented Reading + +- Suggestion never starts: availability evaluator, focus capability, settings snapshot. +- Suggestion uses text from before the keypress: host-publish polling and focus refresh timing. +- Old text appears after switching fields: work ID and focus/content signature guards. +- Partial appears then vanishes: final normalization, confidence, or seam suppression. +- Tab passes through despite visible text: overlay/session acceptance validation and input tap state. +- A rapid second Tab escapes after the tail is exhausted: post-exhaustion arm, queued accept, and + generation-keyed backstop. +- Accepted word repeats: post-insertion reconciliation and speculative-generation validation. +- Typo correction behaves like a continuation: session kind and correction acceptance path. +- Ghost tail jitters after partial acceptance: acceptance presentation and overlay advance logic. + +## Update This Guide When + +Update this document when a new pipeline state is introduced, a stale-result signature changes, +correction ordering moves, request construction gains a context source, streaming semantics change, +or acceptance/session reconciliation acquires a new invariant. diff --git a/.internal/interview-prep/README.md b/.internal/interview-prep/README.md new file mode 100644 index 00000000..322a1e52 --- /dev/null +++ b/.internal/interview-prep/README.md @@ -0,0 +1,802 @@ +# Cotabby Interview Study Path + +## Purpose + +This is an untimed, mastery-based study path for explaining Cotabby in a technical interview. It +does not repeat the detailed architecture guides. It tells you what to read, which concrete symbols +to trace, what decisions to understand, and what you must be able to reconstruct without notes. + +The objective is not to memorize the repository. The objective is to internalize the product as a +set of reliability boundaries: + +~~~text +process lifecycle + -> focused editor truth + -> global input intent + -> eligible immutable request + -> selected generation backend + -> normalized active session + -> non-activating presentation + -> validated insertion + -> host publication reconciliation +~~~ + +You are ready when you can enter the repository through a symptom or design question, name the +responsible owner, trace the data flow, explain the tradeoff, and identify the invariant that keeps +the user safe. + +## Source-of-Truth Order + +Use documentation in this order: + +1. [Root architecture map](../../ARCHITECTURE.md) for the whole-system mental model. +2. The relevant guide under [architecture](../architecture/) for the subsystem explanation. +3. The linked production source for current behavior. +4. Tests for executable examples and edge cases. + +[README.md](../../README.md) is the product-facing overview, while [AGENTS.md](../../AGENTS.md) is the +canonical coding-agent instruction file. Both are synchronized with the three-engine architecture, +but neither replaces the subsystem guides or production source for implementation-level questions. +When any prose conflicts with code and tests, verify the owner directly and fix the documentation. + +## How to Study + +For each section: + +1. Read the named architecture guide. +2. Open the production files in the stated order. +3. Locate the named symbols rather than scrolling randomly. +4. Draw the input, mutable owner, async boundary, output, and stale-result guard. +5. Complete the trace exercise without copying code. +6. Answer the checkpoint questions aloud. +7. Reopen the code only after you have committed to an answer. + +When explaining a decision, use this shape: + +~~~text +constraint + -> chosen boundary or mechanism + -> failure it prevents + -> cost or tradeoff + -> evidence in specific files + -> improvement you would consider +~~~ + +That format demonstrates engineering judgment. Merely describing the implementation demonstrates +code familiarity but not architectural understanding. + +## Mastery Area 1: Product and System Shape + +### Read + +- [Root architecture map](../../ARCHITECTURE.md) +- [Lifecycle and Composition](../architecture/lifecycle-and-composition.md) +- [Suggestion Pipeline](../architecture/suggestion-pipeline.md) + +### Open + +1. [CotabbyApp.swift](../../Cotabby/App/Core/CotabbyApp.swift) +2. [AppDelegate.swift](../../Cotabby/App/Core/AppDelegate.swift) +3. [CotabbyAppEnvironment.swift](../../Cotabby/App/Core/CotabbyAppEnvironment.swift) +4. [SuggestionCoordinator.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator.swift) +5. [SuggestionSettingsModel.swift](../../Cotabby/Models/Settings/SuggestionSettingsModel.swift) +6. [SuggestionSettingsData.swift](../../Cotabby/Models/Settings/SuggestionSettingsData.swift) + +### Find + +- CotabbyApp.body +- AppDelegate.applicationDidFinishLaunching +- AppDelegate.applicationWillTerminate +- CotabbyAppEnvironment.init +- SuggestionCoordinator dependencies, internal state, and the small UI-observed published surface +- SuggestionSettingsModel.domainSettings and snapshot + +### Understand + +Cotabby is not primarily a text-generation application. It is a cross-process editor integration +whose model call sits in the middle of a longer state machine. Reliability depends on focus truth, +event integrity, stale-work rejection, overlay ownership, insertion correctness, and host +reconciliation. + +CotabbyApp declares SwiftUI scenes. CotabbyAppEnvironment constructs one shared dependency graph. +AppDelegate controls when app-scoped side effects begin and end. This prevents duplicate AX polling, +event taps, runtime managers, panels, and settings models. + +Settings have one durable owner. SwiftUI continues to bind the model's individual published +properties, `domainSettings` projects them into cohesive product areas, the snapshot freezes the +behavior subset for async work, and the store preserves flat UserDefaults keys. + +### Trace Exercise + +Draw process startup from the App entry point through environment construction to these services: + +- FocusTracker +- InputMonitor +- SuggestionCoordinator +- VisualContextCoordinator +- LlamaRuntimeManager +- InlineCommandCoordinator + +For each one, distinguish construction from startup. Then reverse the drawing for process +termination. + +### Checkpoints + +- Why is constructing a service different from starting it? +- Which subscriptions belong to CotabbyAppEnvironment, and which belong to AppDelegate? +- Why would constructing FocusTracker or InputMonitor inside a SwiftUI view be dangerous? +- Why does the XCTest host skip production startup? +- What resources must be stopped before native runtime shutdown? +- Give a one-minute description of Cotabby without leading with llama.cpp. + +## Mastery Area 2: Focus as Eventually Consistent State + +### Read + +- [Focus and Accessibility](../architecture/focus-and-accessibility.md) +- The privacy boundary in [Context, Privacy, and Permissions](../architecture/context-privacy-and-permissions.md) + +### Open + +1. [FocusTracker.swift](../../Cotabby/Services/Focus/FocusTracker.swift) +2. [FocusSnapshotResolver.swift](../../Cotabby/Services/Focus/FocusSnapshotResolver.swift) +3. [FocusModels.swift](../../Cotabby/Models/Focus/FocusModels.swift) +4. [AXTextGeometryResolver.swift](../../Cotabby/Services/Focus/AXTextGeometryResolver.swift) +5. [AXHelper.swift](../../Cotabby/Support/Accessibility/AXHelper.swift) + +### Find + +- FocusTracker.start, refreshNow, performCaptureAndPublish, and resolveChromiumFocusFallback +- FocusSnapshotResolver.resolveSnapshot, resolveCandidate, boundedContextWindow, and candidateSnapshot +- FocusCapability and FocusedInputSnapshot +- CaretGeometryQuality +- AXTextGeometryResolver.resolveCaretRect and its range, marker, character, static-run, and field + fallback branches + +### Understand + +AX is synchronous cross-process IPC. Different applications expose different roles, selection +representations, text ranges, and geometry. A FocusSnapshot is therefore a normalized observation, +not permanent truth. + +Polling is authoritative because AXObserver coverage and ordering are inconsistent. Explicit refresh +is still a complete capture, not trust in a notification payload. Adaptive backoff reduces idle cost. +Chromium accessibility priming and hit testing recover browser cases that the system-wide focused +element query misses. + +Identity is deliberately layered. Element identifiers can be recycled, so focusChangeSequence, +process identity, content signatures, and selection are used according to the downstream invariant. + +### Trace Exercise + +Trace a focused Gmail editor from FocusTracker.performCaptureAndPublish through +FocusSnapshotResolver.resolveSnapshot. Record: + +- How the owning process is determined +- How editable candidates are ranked +- Where text is bounded +- Where secure capability is decided +- How caret geometry receives a quality +- Which values become FocusedInputSnapshot +- What causes a new focusChangeSequence + +Repeat conceptually for a native NSTextView and note which compatibility branches disappear. + +### Checkpoints + +- Why is NSWorkspace.frontmostApplication insufficient? +- Why is an editable AX role insufficient? +- Why bound text before publishing the snapshot rather than only in the prompt renderer? +- What makes exact, derived, estimated, and layoutEstimated geometry different? +- Why does presentation care about geometry quality? +- What can go wrong if a cache is keyed only by elementIdentifier? +- Why are AX calls kept on MainActor despite their latency risk? +- What is the current secure-field acquisition limitation? + +## Mastery Area 3: Global Input Without Stealing Keystrokes + +### Read + +- [Input and Insertion](../architecture/input-and-insertion.md) +- Input handling in [Suggestion Pipeline](../architecture/suggestion-pipeline.md) + +### Open + +1. [InputMonitor.swift](../../Cotabby/Services/Input/InputMonitor.swift) +2. [InputSuppressionController.swift](../../Cotabby/Services/Input/InputSuppressionController.swift) +3. [KeyboardInputSourceMonitor.swift](../../Cotabby/Services/Input/KeyboardInputSourceMonitor.swift) +4. [InlineCommandCoordinator.swift](../../Cotabby/App/Coordinators/InlineCommandCoordinator.swift) + +### Find + +- InputMonitor.start and refresh +- installObserverTapIfNeeded +- installAcceptTapIfNeeded +- installToggleTapIfNeeded +- updateAcceptTapState +- handleObserverKeyDown +- handleAcceptKeyDown +- acceptanceKind +- InputMonitorAcceptTapDecision + +### Understand + +The steady observer is listen-only and cannot consume user input. A separate default tap exists only +while a visible suggestion or inline-command capture needs interception. Even then, a matching key is +consumed only after the current owner returns success. Stale or declined ownership passes the +original event through. + +The global toggle has independent lifetime because it must work without a visible suggestion. +InlineCommandCoordinator arbitrates the one capture slot between emoji and macros. + +Synthetic insertion events are tagged and covered by bounded suppression so Cotabby does not treat +its own writes as new user typing. Suppression must expire quickly enough that real typing is never +hidden. + +### Trace Exercise + +Trace one configured word-accept key twice: + +1. A valid visible suggestion exists and acceptance succeeds. +2. The consuming tap still exists but the session became stale before the key arrived. + +Show exactly why the first event is swallowed and the second reaches the host application. + +Then trace one Unicode event posted by SuggestionInserter and explain why neither the listen-only +observer nor the consuming accept tap should treat it as user intent. + +### Checkpoints + +- Why not install one permanently consuming event tap? +- What does fail-open mean here? +- Why does key recognition read current settings at event time? +- Why does the accept tap linger briefly after the overlay hides? +- Why does a synthetic source marker exist in addition to a suppression count? +- How can emoji use the accept key without racing suggestion acceptance? +- What should happen if Input Monitoring permission is revoked while a suggestion is visible? + +## Mastery Area 4: Scheduling and Building a Request + +### Read + +- [Suggestion Pipeline](../architecture/suggestion-pipeline.md) +- [Context, Privacy, and Permissions](../architecture/context-privacy-and-permissions.md) + +### Open + +1. [SuggestionCoordinator+Input.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator+Input.swift) +2. [SuggestionCoordinator+Prediction.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator+Prediction.swift) +3. [SuggestionWorkController.swift](../../Cotabby/Services/Suggestion/SuggestionWorkController.swift) +4. [SuggestionAvailabilityEvaluator.swift](../../Cotabby/Support/Suggestion/SuggestionAvailabilityEvaluator.swift) +5. [SuggestionRequestFactory.swift](../../Cotabby/Support/Suggestion/SuggestionRequestFactory.swift) +6. [SuggestionRequest.swift](../../Cotabby/Models/Suggestion/SuggestionRequest.swift) + +### Find + +- handleFocusSnapshotChange +- handleInputEvent +- schedulePredictionAfterHostPublishDelay +- pollForHostPublish +- schedulePrediction +- generateFromCurrentFocus +- dispatchGeneration +- SuggestionWorkController.replaceDebouncedWork, replaceGenerationWork, isCurrent, and cancelAll +- SuggestionAvailabilityEvaluator.disabledReason +- SuggestionRequestFactory.buildRequest + +### Understand + +A global keydown often arrives before the host publishes its new AX value. The coordinator therefore +does not immediately assume the snapshot contains the typed character. It debounces, requests fresh +focus state, and performs bounded host-publication polling. + +Cancellation is advisory. Native or system work may complete after cancellation, so every unit of +prediction work also carries a monotonically increasing work ID. Result application validates that +ID plus focus, content, session, and settings continuity. + +SuggestionRequestFactory separates the question of what to request from when to request. It receives +already captured values, applies engine-specific bounds and optional-context budgets, and returns an +immutable SuggestionRequest with a request ID. + +### Trace Exercise + +Trace the typed character a from InputMonitor through: + +- host-publication delay +- availability evaluation +- work replacement +- fresh focus materialization +- correction gate +- clipboard relevance +- visual excerpt lookup +- request construction +- engine dispatch + +At each await, write the condition that could make the work stale. + +### Checkpoints + +- Why are cancellation and work identity both required? +- Why gate eligibility more than once? +- Why does an engine receive an immutable request rather than a focus service? +- Which optional contexts can enter a request? +- Which layer owns context acquisition, and which owns prompt budgeting? +- Why does the caret prefix receive priority over optional context? +- Why does every request need a correlation ID? +- What is speculative post-acceptance generation, and how is an incorrect speculation rejected? + +## Mastery Area 5: Streaming, Sessions, and Reconciliation + +### Read + +- Streaming, normalization, and sessions in [Suggestion Pipeline](../architecture/suggestion-pipeline.md) +- Presentation stability in [Presentation and Sibling Features](../architecture/presentation-and-sibling-features.md) + +### Open + +1. [SuggestionCoordinator+Prediction.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator+Prediction.swift) +2. [SuggestionInteractionState.swift](../../Cotabby/Services/Suggestion/SuggestionInteractionState.swift) +3. [SuggestionSessionReconciler.swift](../../Cotabby/Support/Suggestion/SuggestionSessionReconciler.swift) +4. [SuggestionTextNormalizer.swift](../../Cotabby/Support/Suggestion/SuggestionTextNormalizer.swift) +5. [CompletionSeamGuard.swift](../../Cotabby/Support/Suggestion/CompletionSeamGuard.swift) +6. [StreamedGhostTextPolicy.swift](../../Cotabby/Support/Suggestion/StreamedGhostTextPolicy.swift) +7. [SuggestionStreamingState.swift](../../Cotabby/Support/Suggestion/SuggestionStreamingState.swift) + +### Find + +- queueStreamedPartial, drainStreamedPartial, and applyStreamedPartial +- SuggestionStreamingState.beginGeneration, enqueue, drain, canRender, and clearSession +- apply(result:workID:) +- SuggestionInteractionState.startSession and reconcileActiveSession +- advanceIfTypedCharactersMatch +- SuggestionSessionReconciler.reconcile +- nextAcceptanceChunk and nextAcceptancePhrase +- SuggestionTextNormalizer.normalizeDetailed + +### Understand + +Streamed partials are cumulative and provisional. SuggestionStreamingState makes pending partials +latest-wins, permits only one scheduled drain per runloop window, and remembers the monotonic rendered +prefix. The coordinator still owns scheduling, freshness checks, session mutation, and AppKit. A +partial can become a real active session so the user can accept before decoding ends, but the final +result can replace or suppress it. + +SuggestionInteractionState owns mutable session facts. SuggestionSessionReconciler owns pure +comparison rules. This split lets acceptance, type-through, trailing-text checks, CJK segmentation, +and known post-insertion lag be tested without running AX or AppKit. + +### Trace Exercise + +Start with the visible suggestion: + +~~~text + meeting tomorrow at 10 +~~~ + +Trace these independent cases: + +- The user types the exact leading space and m +- The user types a divergent character +- The user accepts one word +- The user accepts the full tail +- AX briefly reports the pre-insertion value +- The final model result is rejected after a partial was displayed + +Identify which owner changes session state and which pure rule decides the transition. + +### Checkpoints + +- Why can a streamed partial become accept-ready? +- Why must the final result remain authoritative? +- What is the invariant between visible overlay text and remaining session text? +- Why is post-insertion AX tolerance represented by a narrow sentinel? +- How does exact type-through avoid unnecessary regeneration? +- Why are corrections represented as a different session kind? +- Where do language-specific acceptance rules belong? + +## Mastery Area 6: Presentation and Insertion + +### Read + +- [Presentation and Sibling Features](../architecture/presentation-and-sibling-features.md) +- [Input and Insertion](../architecture/input-and-insertion.md) + +### Open + +1. [SuggestionOverlayPresenter.swift](../../Cotabby/Services/Suggestion/SuggestionOverlayPresenter.swift) +2. [OverlayController.swift](../../Cotabby/Services/Presentation/OverlayController.swift) +3. [CompletionRenderModePolicy.swift](../../Cotabby/Support/Presentation/CompletionRenderModePolicy.swift) +4. [SuggestionOverlayStabilityGate.swift](../../Cotabby/Support/Presentation/SuggestionOverlayStabilityGate.swift) +5. [SuggestionCoordinator+Acceptance.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator+Acceptance.swift) +6. [SuggestionInserter.swift](../../Cotabby/Services/Suggestion/SuggestionInserter.swift) +7. [InsertionSafetyGate.swift](../../Cotabby/Support/Suggestion/InsertionSafetyGate.swift) +8. [InsertionStrategySelector.swift](../../Cotabby/Support/Suggestion/InsertionStrategySelector.swift) +9. [PostExhaustionAcceptanceState.swift](../../Cotabby/Support/Suggestion/PostExhaustionAcceptanceState.swift) + +### Find + +- SuggestionOverlayPresenter.present +- OverlayController.showSuggestion, showInline, showMirror, and advanceInline +- CompletionRenderModePolicy.mode +- acceptCurrentSuggestion, acceptEntireSuggestion, and acceptEnabledSuggestion +- presentAdvancedOverlay and schedulePostInsertionRefresh +- armPostExhaustionAcceptance and flushQueuedPostExhaustionAcceptIfNeeded +- SuggestionInserter.insert, replace, insertViaPaste, and pressPasteMenuItem + +### Understand + +The panel is borderless, non-activating, mouse-ignoring, and space/full-screen compatible. AppKit owns +window behavior; SwiftUI describes its contents. + +Automatic mode paints inline only with exact or derived geometry and a safe end-of-line seam. +Estimated, layout-estimated, and mid-line cases use a mirror card. This avoids visually pretending +that approximate geometry identifies an exact glyph position. + +Short ordinary insertions use Unicode CGEvents. A composing IME uses paste because a Unicode event +can re-enter composition. Optional long/multiline paste snapshots every pasteboard representation, +tries the target application's AX Paste menu item, falls back to Command-V, and restores only if the +clipboard has not changed. + +### Trace Exercise + +Trace a partial word acceptance from acceptCurrentSuggestion through session preparation, insertion, +overlay advance, post-insertion sentinel, focus refresh, and session reconciliation. + +Repeat for: + +- A Japanese IME-active field +- A Chrome field where synthetic Command-V fails +- A correction that replaces a typo +- A user clipboard change during the restore delay + +### Checkpoints + +- Why does OverlayController own NSPanel instead of SuggestionCoordinator? +- Why is layoutEstimated still a mirror-card quality? +- Why can the overlay advance without waiting for fresh AX geometry? +- What protects the user's clipboard during overlapping paste insertions? +- Why is posting a CGEvent not proof that insertion succeeded? +- Why must acceptance validate both the session and visible overlay? +- Why can rapid Tab queue only one unseen accept while an exhausted tail regenerates? +- How does the generation-keyed backstop guarantee Tab ownership returns to the host? +- What is the safest behavior when any acceptance precondition fails? + +## Mastery Area 7: The Three Generation Backends + +### Read + +- [Inference and Prompting](../architecture/inference-and-prompting.md) +- Engine privacy in [Context, Privacy, and Permissions](../architecture/context-privacy-and-permissions.md) + +### Open + +1. [SuggestionEngineRouter.swift](../../Cotabby/Services/Runtime/SuggestionEngineRouter.swift) +2. [FoundationModelSuggestionEngine.swift](../../Cotabby/Services/Runtime/FoundationModelSuggestionEngine.swift) +3. [LlamaSuggestionEngine.swift](../../Cotabby/Services/Runtime/LlamaSuggestionEngine.swift) +4. [LlamaRuntimeManager.swift](../../Cotabby/Services/Runtime/LlamaRuntimeManager.swift) +5. [LlamaRuntimeCore.swift](../../Cotabby/Services/Runtime/LlamaRuntimeCore.swift) +6. [OpenAICompatibleSuggestionEngine.swift](../../Cotabby/Services/Runtime/OpenAICompatibleSuggestionEngine.swift) +7. [OpenAICompatibleAPIClient.swift](../../Cotabby/Services/Runtime/OpenAICompatibleAPIClient.swift) +8. [BaseCompletionPromptRenderer.swift](../../Cotabby/Support/Prompting/BaseCompletionPromptRenderer.swift) +9. [FoundationModelPromptRenderer.swift](../../Cotabby/Support/Prompting/FoundationModelPromptRenderer.swift) + +### Find + +- SuggestionEngineRouter.generateSuggestion, prewarm, resetCachedGenerationContext, and + generateOpenSourceFallback +- FoundationModelSuggestionEngine.ensureSession +- LlamaSuggestionEngine.generateSuggestion and resetCachedGenerationContext +- LlamaRuntimeManager.prepare, generate, stop, stopAndWait, and shutdownSync +- LlamaRuntimeCore.prepare, generate, preparedPrompt, obtainAutocompleteSequence, + resetPromptCache, and shutdown +- OpenAICompatibleSuggestionEngine.generateSuggestion and prewarm +- BaseCompletionPromptRenderer.prompt +- FoundationModelPromptRenderer.sessionInstructions and prompt + +### Understand + +The router provides one generation contract while preserving backend-specific lifecycle: + +- Apple uses an instruction channel and one-use compatible prewarmed session. +- Llama uses a base completion prompt, in-process native pointers, prompt token/KV reuse, token + streaming, and explicit shutdown. +- An OpenAI-compatible endpoint uses completion or chat request transport and SSE parsing. It may be + loopback, LAN, or public HTTPS. + +LlamaRuntimeManager is MainActor and publishes user-facing state. LlamaRuntimeCore is a nonisolated, +lock/condition-protected class that owns native correctness. It is not a Swift actor. The +autocomplete lock serializes prompt-cache/decode state; lifecycle coordination protects load, +active operations, abort, and shutdown. + +### Trace Exercise + +Take one SuggestionRequest and trace it separately through all three engines. For each path identify: + +- Prompt shape +- Prewarm behavior +- Streaming mechanism +- Cancellation mechanism +- Cache or session state +- Output normalization +- Error classification +- Privacy boundary +- Cleanup behavior + +Then trace an engine switch from Open Source to endpoint and explain why the llama runtime is stopped. + +### Checkpoints + +- Why does Apple have a narrow fallback to llama rather than universal silent fallback? +- Why are base GGUF prompts continuation-shaped? +- Why does the caret prefix come last? +- Why is LlamaRuntimeCore not simply MainActor-isolated? +- Why use explicit locking instead of casually wrapping native pointers in Task calls? +- What is safe KV-cache reuse? +- Why broadcast context reset to every backend? +- What data can leave the Mac in endpoint mode? +- Why reject insecure public HTTP but allow loopback HTTP? + +## Mastery Area 8: Context, Permissions, and Honest Privacy + +### Read + +- [Context, Privacy, and Permissions](../architecture/context-privacy-and-permissions.md) + +### Open + +1. [PermissionManager.swift](../../Cotabby/Services/Permission/PermissionManager.swift) +2. [PermissionGuidanceController.swift](../../Cotabby/Services/Permission/PermissionGuidanceController.swift) +3. [ClipboardContextProvider.swift](../../Cotabby/Services/Context/ClipboardContextProvider.swift) +4. [ClipboardRelevanceFilter.swift](../../Cotabby/Support/Context/ClipboardRelevanceFilter.swift) +5. [VisualContextCoordinator.swift](../../Cotabby/Services/Visual/VisualContextCoordinator.swift) +6. [WindowScreenshotService.swift](../../Cotabby/Services/Visual/WindowScreenshotService.swift) +7. [ScreenTextExtractor.swift](../../Cotabby/Services/Visual/ScreenTextExtractor.swift) +8. [OCRTextHygiene.swift](../../Cotabby/Support/Context/OCRTextHygiene.swift) +9. [PromptContextSanitizer.swift](../../Cotabby/Support/Context/PromptContextSanitizer.swift) +10. [OpenAICompatibleEndpointModels.swift](../../Cotabby/Models/Runtime/OpenAICompatibleEndpointModels.swift) + +### Find + +- PermissionManager.refresh and requiredPermissionsGranted +- ClipboardRelevanceFilter.filter +- VisualContextCoordinator.startSessionIfNeeded, launchSession, and applyExcerpt +- ScreenshotContextGenerator.generateContext and finishedExcerpt +- OCRTextHygiene.clean +- OpenAICompatibleEndpointConfiguration validation and privacyWarning +- SuggestionAvailabilityEvaluator.shouldCaptureVisualContext + +### Understand + +Accessibility and Input Monitoring are required for core autocomplete. Screen Recording is optional. +Visual context is a field-scoped screenshot-to-Vision-OCR pipeline. OCR confidence and hygiene remove +noise; no model summarization occurs; only a bounded sanitized excerpt can reach a prompt. + +Clipboard context is read at request time, relevance-filtered, distilled, and not retained as a +history. User-authored context, surface metadata, and visual context have independent budgets. + +Privacy claims must distinguish on-device Apple/llama generation from a configured endpoint. They +must also describe the current secure-field limitation honestly: generation and insertion are +blocked, but a bounded FocusedInputSnapshot is still created and visual capture can run because the +visual eligibility gate ignores capability. + +### Trace Exercise + +Build a context inventory for one request. For each field record: + +- Acquisition owner +- Permission +- In-memory lifetime +- First bound +- Prompt budget +- Whether it can be logged in debug mode +- Whether it leaves the Mac under each engine + +Then trace a secure field far enough to show where assistance stops and where acquisition currently +does not. + +### Checkpoints + +- Why is Screen Recording optional? +- Why is visual context scoped to a field instead of regenerated on every key? +- Why is raw OCR cleaned rather than summarized by another model? +- Why does relevance filtering matter for clipboard context? +- What exactly does local-first mean in the current product? +- Where are endpoint credentials stored? +- Which privacy claim would currently be false? +- How would you move toward a true no-secure-field-acquisition invariant? + +## Mastery Area 9: Observability, Tests, and Failure Diagnosis + +### Read + +- Debugging and validation in [Root architecture map](../../ARCHITECTURE.md) +- Failure-oriented sections in every architecture guide + +### Open + +1. [CotabbyDebugOptions.swift](../../Cotabby/Support/Logging/CotabbyDebugOptions.swift) +2. [RequestID.swift](../../Cotabby/Support/Logging/RequestID.swift) +3. [SuggestionDebugLogger.swift](../../Cotabby/Services/Suggestion/SuggestionDebugLogger.swift) +4. [FileLogHandler.swift](../../Cotabby/Support/Logging/FileLogHandler.swift) +5. [LLMIOFileHandler.swift](../../Cotabby/Support/Logging/LLMIOFileHandler.swift) +6. [CotabbyTests](../../CotabbyTests) +7. [build workflow](../../.github/workflows/build.yml) +8. [test workflow](../../.github/workflows/tests.yml) +9. [XcodeGen workflow](../../.github/workflows/xcodegen.yml) +10. [lint workflow](../../.github/workflows/lint.yml) + +### Understand + +A request ID joins coordinator stages, backend selection, generation, performance, acceptance, and +debug LLM I/O. The always-on path uses unified logging. Explicit -cotabby-debug enables local JSONL, +full prompts/completions, AX dumps, and visual capture artifacts. + +Tests are strongest around deterministic Support and Models rules. Stateful owners expose narrow +protocols and fakes so work identity, session transitions, trigger machines, prompt rendering, and +layout policy can run without global permissions. + +XcodeGen makes project.yml the project source of truth. CI intentionally treats a regenerated +project diff as a failure. + +### Trace Exercise + +For each symptom, identify the first log category, correlation key, production file, and pure test +surface: + +- A suggestion from the previous field appears +- Tab is swallowed without insertion +- A Chrome caret is one line too low +- Llama remains resident after switching engines +- Visual context belongs to the previous field +- A final result suppresses a streamed partial + +### Checkpoints + +- Why is a request ID more useful than chronological logs alone? +- Which debug artifacts can contain private text? +- What should be unit tested without launching the app? +- What requires a real host-application compatibility test? +- Why are build, tests, lint, and XcodeGen separate CI checks? +- What would you instrument before trying to fix an intermittent editor bug? + +## The Six Golden Traces + +You should be able to draw these from memory and name the files at every arrow. + +### Trace 1: Process Startup + +~~~text +CotabbyApp + -> AppDelegate.init + -> CotabbyAppEnvironment.init + -> applicationDidFinishLaunching + -> runtime / focus / input / suggestion / inline command startup +~~~ + +### Trace 2: Focus Acquisition + +~~~text +FocusTracker poll + -> focused AX element or browser fallback + -> FocusSnapshotResolver + -> bounded FocusedInputSnapshot + capability + -> FocusTrackingModel publication + -> SuggestionCoordinator focus reaction +~~~ + +### Trace 3: Typed Character to Streamed Ghost Text + +~~~text +listen-only input event + -> host-publication refresh + -> availability + -> work ID + -> request factory + -> engine router + -> normalized cumulative partial + -> active session + -> overlay presenter +~~~ + +### Trace 4: Word Acceptance + +~~~text +conditional consuming tap + -> session + overlay validation + -> next acceptance chunk + -> SuggestionInserter + -> synchronous tail advance + -> post-insertion AX sentinel + -> fresh snapshot reconciliation + -> bounded post-exhaustion ownership if the tail ended +~~~ + +### Trace 5: Field Switch During Generation + +~~~text +new focus sequence + -> cancel old tasks + -> increment work identity + -> reset backend context + -> clear old session / overlay + -> late old result rejected by current-work and focus/content checks +~~~ + +### Trace 6: Engine Switch + +~~~text +settings/profile selection + -> cancel prediction + -> reset cached generation context + -> router selects backend + -> AppDelegate starts or stops llama runtime + -> selected backend prewarm +~~~ + +## Interview Readiness Checklist + +### Product + +- Explain Cotabby in one minute without reducing it to an LLM wrapper. +- State the required and optional permissions. +- Explain local-first without falsely claiming every engine is offline. +- Name the supported presentation and acceptance modes. + +### Ownership + +- Identify the composition root and lifecycle owner. +- Explain why views do not construct process-wide services. +- Distinguish coordinator orchestration, service side effects, model state, and pure Support rules. +- Explain why both environment and AppDelegate retain subscriptions. + +### Reliability + +- Explain focus eventual consistency. +- Explain cancellation plus work identity. +- Explain focus/content/session revalidation after awaits. +- Explain fail-open input consumption. +- Explain post-insertion reconciliation. +- Explain streamed partial versus authoritative final. +- Explain latest-wins stream draining and the bounded post-exhaustion rapid-accept window. + +### macOS + +- Explain the roles of Accessibility, Input Monitoring, and Screen Recording. +- Explain why AppKit panels are needed next to SwiftUI. +- Explain browser/Electron compatibility work. +- Explain IME-safe insertion and clipboard restoration. +- Explain TCC identity and why Cotabby Dev is separate. + +### Inference + +- Compare Apple, llama, and endpoint prompt/stream/lifecycle behavior. +- Explain manager versus core ownership. +- Explain safe prompt/KV reuse. +- Explain backend-independent normalization. +- Explain when data can leave the machine. + +### Critical Judgment + +- Identify current architecture strengths. +- Identify current complexity or debt without being dismissive. +- Explain the secure-field acquisition limitation honestly. +- Propose a measured improvement rather than a rewrite. +- Translate the invariants into a reliability plan for HyperWrite. + +## Final Active-Recall Drill + +Close the codebase and answer: + +1. What are the three most dangerous races in Cotabby? +2. What are the three ways Cotabby could accidentally interfere with another application? +3. Which owner is responsible for preventing each one? +4. Where does untrusted or stale information become a stable domain value? +5. Which state is app-scoped, field-scoped, request-scoped, session-scoped, and token-stream-scoped? +6. What happens when the host never publishes the insertion Cotabby expected? +7. What happens when the user switches engines during decode? +8. Which work stays on MainActor and which work must leave it? +9. Which claims are product promises, and which are current implementation limitations? +10. Which Cotabby patterns should transfer to HyperWrite, and which should be reconsidered? + +If an answer is vague, return to the named trace rather than rereading the repository from the +beginning. diff --git a/.internal/interview-prep/hyperwrite-reliability-translation.md b/.internal/interview-prep/hyperwrite-reliability-translation.md new file mode 100644 index 00000000..e11bc684 --- /dev/null +++ b/.internal/interview-prep/hyperwrite-reliability-translation.md @@ -0,0 +1,878 @@ +# Translating Cotabby into a Reliable HyperWrite Mac Alpha + +## Purpose + +This document prepares you to discuss how Cotabby's lessons apply to the HyperWrite Mac prototype. +It is not a claim about HyperWrite's current implementation; you have not inspected that prototype +yet. It separates: + +- Questions that must be answered during the technical session +- Reliability invariants that apply to almost any cross-application Mac writing assistant +- Cotabby patterns worth transferring +- Cotabby-specific decisions that should not be copied blindly +- A practical sequence for turning a prototype into an alpha users can trust + +The goal is to show Josh that you can productize an existing prototype without prematurely rewriting +it or treating model integration as the whole problem. + +## The Core Interview Thesis + +A strong opening position is: + +> I would treat HyperWrite Mac as a cross-application interaction state machine, not just an API +> client with an overlay. The user has to trust that we understand the current editor, never show a +> stale suggestion, never steal an unrelated key, and insert exactly what was accepted. I would first +> instrument the prototype and define those invariants, then harden focus, input, session, insertion, +> and failure recovery around a declared compatibility matrix. + +That communicates three things: + +1. You understand where Mac-wide autocomplete fails in production. +2. You will learn from the prototype before replacing it. +3. You define reliability in observable behavior rather than general confidence. + +## Reliability Is a Product Contract + +For this product, reliable should mean: + +### Input integrity + +- HyperWrite never consumes a key it did not successfully handle. +- Synthetic insertion never re-enters the suggestion pipeline as user typing. +- Shortcut changes take effect predictably. +- An active IME does not cause acceptance to disappear into composition. + +### Target correctness + +- A suggestion belongs to one focused field and one text seam. +- Switching applications, fields, selections, or documents invalidates incompatible work. +- A late network or model response cannot appear in a newer editor state. + +### Session correctness + +- Visible suggestion text and the active remaining tail agree. +- Type-through advances only on exact matching characters. +- Partial acceptance inserts one deterministic chunk. +- A final stream result cannot silently corrupt an already accepted session. + +### Insertion correctness + +- Accepted text is committed through a strategy appropriate to the host and input method. +- The original acceptance key is consumed only when the write path succeeds. +- The host's subsequent state is reconciled; posting an event is not considered proof. +- Clipboard fallbacks preserve user data and respect newer clipboard changes. + +### Presentation correctness + +- HyperWrite UI never steals focus from the assisted editor. +- Approximate caret geometry is not presented as exact. +- The overlay stays within the correct screen/Space and degrades when positioning is uncertain. +- Hiding or advancing presentation cannot leave an accept-ready invisible session. + +### Availability and recovery + +- Missing permissions, network, authentication, model availability, sleep/wake, or host exit produce + explicit recoverable states. +- Cancellation is normal lifecycle, not an alarming error. +- Failed prewarm or optional context does not necessarily disable the core loop. +- Shutdown cannot race active native or network work. + +### Privacy and security + +- Every acquired context source has a permission, lifetime, bound, transport scope, and disclosure. +- Secure fields are rejected at the earliest practical acquisition boundary. +- Credentials use platform secret storage. +- Debug artifacts containing user text require an explicit mode and retention policy. + +### Observability + +- One suggestion can be traced from focus to request to stream to presentation to acceptance. +- Failures are classified by stage and reason rather than flattened into “did not work.” +- Metrics distinguish unsupported hosts from product regressions. + +## What Is Known and What Is Not + +Known from Josh's message: + +- There is a current HyperWrite Mac prototype. +- The objective is a strong Mac alpha. +- The technical session is intended to examine architecture and approach. +- Reliability and longer-term engineering fit are part of the evaluation. + +Unknown until the session: + +- Whether the app is Swift/AppKit/SwiftUI, Electron, Catalyst, or mixed +- How it discovers focused editors +- Whether it uses AX polling, notifications, event taps, or application-specific integration +- Which keys it observes or consumes +- Whether generation is local, hosted, hybrid, or streamed +- How request identity and cancellation work +- How it positions and owns its overlay +- How it inserts accepted text +- Which applications and languages are in alpha scope +- What telemetry or diagnostics already exist +- How it is signed, distributed, updated, and granted TCC permissions +- Which prototype failures are already known + +Do not fill these gaps with assumptions. Use them to demonstrate disciplined discovery. + +## Questions to Ask During the Technical Session + +### Product contract + +- What is the smallest user journey that must feel excellent in the alpha? +- Is the primary interaction inline autocomplete, rewrite commands, chat, or several modes? +- What applications are explicitly in scope? +- Are browsers, Electron editors, native fields, Office apps, and IDEs equally important? +- Are multiline, mid-line, and selected-text operations required? +- What are the intended acceptance and dismissal gestures? +- What does Josh mean by “strong alpha”: internal dogfood, invited users, or public release? +- Which current prototype behaviors are most embarrassing or unreliable? + +### Focus and editor state + +- How is the focused editable element discovered today? +- What representation of text, selection, and caret geometry is treated as authoritative? +- How are browser iframes and custom editors handled? +- Are secure/read-only/terminal fields classified? +- Is text bounded before it enters application state? +- What identifies the same field across AX element replacement? +- Is there a known compatibility matrix? + +### Input + +- Does the prototype use CGEvent taps, NSEvent global monitors, local monitors, or another mechanism? +- Is the tap listen-only or capable of consuming events? +- Under what exact condition is an acceptance key swallowed? +- How are synthetic writes distinguished from physical input? +- How do shortcut changes and modifier state interact with active capture? +- What happens when Input Monitoring permission is revoked? + +### Request and generation + +- Where is an immutable request assembled? +- Which context sources can be included? +- Is generation streamed, and are partials cumulative or deltas? +- What backend owns authentication, retries, timeout, and cancellation? +- Can responses arrive out of order? +- Is there request or session identity across client and server logs? +- What quality cleanup happens before display? +- What is the desired behavior when the backend is slow or offline? + +### Presentation + +- Is the overlay an NSPanel, SwiftUI scene, Electron window, or host-integrated view? +- Can it become key or steal focus? +- How is caret geometry obtained and classified? +- What happens when exact geometry is unavailable? +- How are multiple displays, Spaces, full-screen apps, RTL, and multiline text handled? +- Can a visible partial be accepted before the final response? + +### Insertion + +- Does acceptance use Unicode events, AX value mutation, paste, menu commands, or per-app strategies? +- How is the host result verified? +- How are IMEs handled? +- What happens if the host ignores or transforms the inserted text? +- If the clipboard is used, how are all representations restored? +- Are replacements and forward continuations represented differently? + +### Privacy + +- Which user text and screen context leaves the machine? +- What is stored by the client and server? +- Are screenshots or OCR involved? +- How are secure fields excluded? +- Where are credentials stored? +- What does debug logging contain and how long is it retained? +- Which privacy claims are already public? + +### Distribution and operations + +- What macOS versions and hardware are supported? +- Is the app sandboxed? Why or why not? +- How are signing, notarization, entitlements, and updates handled? +- Are development and production TCC identities separate? +- Is there a crash-reporting or support-diagnostics path? +- How are releases rolled back? + +### Code and team + +- Which parts of the prototype are considered sound and should be preserved? +- Where does Josh expect architectural change? +- What tests exist? +- What build/release automation exists? +- Who decides product tradeoffs during the trial? +- What access, designs, backend contracts, and user feedback will be available? + +## How to Inspect the Prototype Live + +Ask Josh to demonstrate one successful suggestion and one known failure. Trace both through the same +questions: + +~~~text +What editor state did the app observe? + -> What event triggered work? + -> What request identity was created? + -> What context was sent? + -> What backend operation ran? + -> What partial/final output returned? + -> What state became accept-ready? + -> What window displayed it? + -> What consumed the acceptance key? + -> What inserted text? + -> How was host success verified? + -> Which logs prove the path? +~~~ + +Do not start with “I would rewrite this.” Start with: + +- Where is the state owned? +- Which invariant is implicit? +- Which boundary lacks identity or observability? +- Is the observed failure deterministic, host-specific, or timing-specific? +- Can the current component be wrapped behind a reliable contract? + +## Provisional Risk Register + +These risks should be validated, not assumed. + +| Risk | User-visible failure | First evidence to seek | Cotabby lesson | +| --- | --- | --- | --- | +| Stale focus | Suggestion appears in the wrong field | Focus/session identifiers in logs | focusChangeSequence plus content signatures | +| AX publish lag | Prompt omits the latest character | Event and AX capture timestamps | bounded host-publication polling | +| Consuming tap ownership | Tab or another key disappears | Tap mode and accept verdict | listen-only observer plus fail-open accept tap | +| Out-of-order network stream | Old partial replaces newer state | Request IDs and stream sequence | work identity and monotonic partial policy | +| Weak caret geometry | Overlay floats or overlaps text | Geometry source/quality | quality-aware inline versus mirror | +| Insertion mismatch | Accepted text is missing or duplicated | Planned write versus fresh host state | insertion strategy plus reconciliation | +| IME composition | Acceptance re-enters composition | Input source and insertion method | IME-aware paste commit | +| Clipboard corruption | User loses clipboard contents | Pasteboard snapshot/restore logs | all-representation, change-aware restore | +| Permission drift | App works after install but not restart | TCC state and code identity | explicit permission model and dev identity | +| Backend outage | UI hangs or stale ghost remains | Timeout/cancel state | recoverable engine state and cancellation | +| Unbounded context | Latency, cost, or privacy leak | Request-size breakdown | acquisition bounds plus section budgets | +| Native/resource leak | Memory grows across sessions | model/window/task lifecycle | selected-runtime lifecycle and bounded shutdown | +| App-specific AX behavior | One host breaks while others pass | compatibility matrix | isolated fallbacks behind focus/geometry services | +| Poor observability | Team cannot reproduce reports | missing correlation/stage metadata | request-correlated structured logs | + +## A Provisional Target Architecture + +The exact types should follow the prototype language and framework, but the responsibilities should +look like this: + +~~~text +ApplicationEnvironment + owns app-lifetime services and configuration + +FocusProvider + reduces host APIs into bounded FocusSnapshot values + +InputMonitor + observes physical intent and conditionally consumes owned actions + +SuggestionCoordinator + owns state transitions, not low-level OS or model mechanics + +WorkController + owns debounce, cancellation, and current work identity + +ContextBuilder + builds one immutable bounded request + +SuggestionEngine + streams backend results behind one contract + +OutputNormalizer + enforces backend-independent display and insertion policy + +InteractionSession + owns active anchor, remaining text, type-through, and acceptance state + +OverlayController + owns non-activating presentation and geometry degradation + +Inserter + selects a host/IME-safe write strategy and reports the plan/result + +Diagnostics + correlates focus, request, stream, presentation, and acceptance +~~~ + +This is not a demand for eleven classes. It is a responsibility map. Several can begin as small value +types or protocols around working prototype code. + +## Cotabby Patterns Worth Transferring + +### One composition root + +Transfer the invariant that process-wide focus monitors, event taps, panels, and sessions have one +owner. The HyperWrite implementation may use dependency injection, an application environment, or +another composition mechanism. + +Cotabby references: + +- [CotabbyAppEnvironment.swift](../../Cotabby/App/Core/CotabbyAppEnvironment.swift) +- [AppDelegate.swift](../../Cotabby/App/Core/AppDelegate.swift) + +### Normalized bounded focus state + +Do not let every subsystem call AX independently. Reduce host state once into a value carrying +capability, text window, selection, identity, geometry quality, and surface metadata. + +Cotabby references: + +- [FocusSnapshotResolver.swift](../../Cotabby/Services/Focus/FocusSnapshotResolver.swift) +- [FocusModels.swift](../../Cotabby/Models/Focus/FocusModels.swift) + +### Explicit work identity + +Cancellation must be paired with a generation/work identifier and environment signatures. Network +responses are especially likely to finish after cancellation. + +Cotabby references: + +- [SuggestionWorkController.swift](../../Cotabby/Services/Suggestion/SuggestionWorkController.swift) +- [SuggestionCoordinator+Prediction.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator+Prediction.swift) + +### A real interaction session + +Represent active suggestion state explicitly. Store its anchor, full text, consumed portion, +trailing-text expectation, and kind. Do not infer the session from whatever the overlay currently +shows. + +Cotabby references: + +- [SuggestionInteractionState.swift](../../Cotabby/Services/Suggestion/SuggestionInteractionState.swift) +- [SuggestionSessionReconciler.swift](../../Cotabby/Support/Suggestion/SuggestionSessionReconciler.swift) + +### Small pure state machines for timing-sensitive mechanisms + +When several booleans and counters protect one timing invariant, move their transitions into a +small value while leaving scheduling and side effects with the coordinator. This makes race rules +executable without pretending the value owns event taps, timers, or windows. + +Cotabby references: + +- [SuggestionStreamingState.swift](../../Cotabby/Support/Suggestion/SuggestionStreamingState.swift) +- [PostExhaustionAcceptanceState.swift](../../Cotabby/Support/Suggestion/PostExhaustionAcceptanceState.swift) +- [SuggestionStreamingStateTests.swift](../../CotabbyTests/Support/Suggestion/SuggestionStreamingStateTests.swift) +- [PostExhaustionAcceptanceStateTests.swift](../../CotabbyTests/Support/Suggestion/PostExhaustionAcceptanceStateTests.swift) + +### Fail-open input + +Keep ordinary observation non-consuming. Enable interception only while a feature owns a key, and +swallow the key only after the action succeeds. + +Cotabby reference: + +- [InputMonitor.swift](../../Cotabby/Services/Input/InputMonitor.swift) + +### Strategy-based insertion + +Treat Unicode events, menu paste, Command-V, AX mutation, and replacements as strategies with +explicit preconditions and failure behavior. Detect composing IMEs. Reconcile afterward. + +Cotabby references: + +- [SuggestionInserter.swift](../../Cotabby/Services/Suggestion/SuggestionInserter.swift) +- [InsertionStrategySelector.swift](../../Cotabby/Support/Suggestion/InsertionStrategySelector.swift) + +### Uncertainty-aware presentation + +Carry geometry quality into presentation. If HyperWrite cannot know an exact caret, use a UI that +looks intentionally approximate instead of misaligned inline text. + +Cotabby references: + +- [CompletionRenderModePolicy.swift](../../Cotabby/Support/Presentation/CompletionRenderModePolicy.swift) +- [OverlayController.swift](../../Cotabby/Services/Presentation/OverlayController.swift) + +### Bounded context at acquisition and rendering + +Bound user text before it circulates through state, then budget it again when creating a request. +Treat each optional context source as a separate privacy boundary. + +Cotabby references: + +- [FocusSnapshotResolver.swift](../../Cotabby/Services/Focus/FocusSnapshotResolver.swift) +- [SuggestionRequestFactory.swift](../../Cotabby/Support/Suggestion/SuggestionRequestFactory.swift) +- [PromptContextSanitizer.swift](../../Cotabby/Support/Context/PromptContextSanitizer.swift) + +### Correlated observability + +Create request/session/focus identifiers early and carry them across client/server if possible. +Record state transitions and suppression reasons, not only errors. + +Cotabby references: + +- [RequestID.swift](../../Cotabby/Support/Logging/RequestID.swift) +- [SuggestionDebugLogger.swift](../../Cotabby/Services/Suggestion/SuggestionDebugLogger.swift) + +## Cotabby Decisions Not to Copy Blindly + +### Polling + +Polling is Cotabby's authoritative choice because of observed AX notification inconsistency. If +HyperWrite already has a reliable notification-plus-reconciliation design or a narrower host set, +measure it before replacing it. The transferable idea is one authoritative freshness model. + +### App-specific fallbacks + +Cotabby has Chromium, Calendar, static-run, and geometry workarounds accumulated from real hosts. +HyperWrite should add compatibility branches only for supported-product evidence, behind a narrow +boundary and regression case. + +### Llama runtime structure + +If HyperWrite is hosted-only, LlamaRuntimeManager/Core and KV-cache lifecycle are irrelevant. The +transferable ideas are serialized mutable backend state, cancellation, prewarm, and cleanup. + +### Coordinator shape + +Do not copy a large coordinator file layout. Copy the distinction between orchestration, mutable +session state, work identity, pure policy, and small mechanism-specific state machines. + +### Every Cotabby feature + +Emoji, macros, visual OCR, power profiles, multiple backends, and extensive settings are not +prerequisites for a reliable HyperWrite alpha. Additional surface area multiplies the compatibility +matrix. + +### Current secure-field acquisition + +Cotabby's secure-field generation block occurs later than the ideal acquisition boundary. HyperWrite +should make the desired privacy invariant explicit before copying focus or visual-context behavior. + +## Work Sequence for an Alpha + +This is sequencing, not a calendar. + +### Phase A: Establish the baseline + +Outputs: + +- Build and run instructions that work from a clean checkout +- A written current architecture and data-flow trace +- A list of known failures with reproduction steps +- Structured stage logging with correlation IDs +- Initial latency, crash, input, and insertion measurements +- A provisional supported-application matrix + +Why first: + +Without a baseline, architectural changes cannot be distinguished from regressions and the team will +optimize the most memorable anecdote. + +### Phase B: Define the interaction contract + +Outputs: + +- FocusSnapshot or equivalent bounded value +- Explicit capability and block reasons +- Request/work identity +- Active interaction session +- Rules for invalidation, type-through, dismissal, and acceptance +- One definition of visible-versus-accept-ready state + +Why: + +This turns implicit timing into testable state transitions. + +### Phase C: Harden focus and input + +Outputs: + +- One authoritative focus freshness mechanism +- Supported-host capability resolution +- Secure/read-only exclusion +- Fail-open input observation/interception +- Synthetic-event suppression +- Permission recovery +- Stress tests for rapid typing, field switching, and modifier changes + +Why: + +No generation improvement matters if the wrong editor is targeted or a physical key disappears. + +### Phase D: Harden request and generation + +Outputs: + +- Immutable bounded request builder +- Backend timeout, cancellation, and retry policy +- Streaming sequence/monotonicity contract +- Backend-independent output normalization +- Explicit unavailable/degraded states +- Client/server request correlation where available + +Why: + +Network/model uncertainty becomes an ordinary state instead of corrupting UI state. + +### Phase E: Harden presentation and insertion + +Outputs: + +- Non-activating overlay ownership +- Geometry quality and fallback presentation +- Session/overlay equality checks +- Host and IME-aware insertion strategy +- Post-insertion verification +- Clipboard protection if paste is used +- Multiple display, Space, full-screen, RTL, and multiline checks appropriate to scope + +Why: + +This is the point where users either trust the product or feel that it interferes with their work. + +### Phase F: Productize distribution and recovery + +Outputs: + +- Clear permission onboarding and recovery +- Stable development and production identities +- Signing/notarization/update path +- Privacy disclosure matching actual context transport +- Debug artifact policy +- Crash/hang and support-diagnostics workflow +- Release rollback plan + +Why: + +A reliable debug build is not yet a reliable product. + +### Phase G: Alpha hardening + +Outputs: + +- Executed compatibility matrix +- Adversarial and soak results +- Measured alpha gates +- Known limitations and non-goals +- Triage playbook +- Prioritized post-alpha roadmap + +Why: + +Alpha quality should be a deliberate support boundary, not “it worked in the demo.” + +## Observability Contract + +Every suggestion should carry identifiers such as: + +- focus_session_id +- request_id +- interaction_session_id +- stream_sequence +- insertion_attempt_id + +Useful stages: + +~~~text +focus_captured +focus_blocked +input_observed +host_publish_wait_started +request_built +request_dispatched +first_partial +partial_presented +final_received +result_suppressed +session_started +accept_requested +insert_attempted +insert_verified +insert_mismatch +session_invalidated +request_cancelled +~~~ + +Each stage should carry only safe metadata by default: + +- Host application and surface classification +- Focus/content signature, not raw text +- Backend and model +- Latency +- Geometry source/quality +- Context character counts by source +- Suppression or failure reason +- Insertion strategy and outcome + +Full text or screenshots should require an explicit diagnostic mode with local retention and clear +consent. If the HyperWrite server already has request IDs, the client request ID should cross the +transport boundary. + +## Failure and Recovery Matrix + +| Failure | Safe behavior | Diagnostic evidence | +| --- | --- | --- | +| No focused supported field | Hide/disable suggestion; consume nothing | capability reason | +| AX state older than input | bounded refresh/poll, then ordinary guards | event and capture ages | +| Field switch during request | cancel and reject late response | request/focus identity mismatch | +| Stream sends non-monotonic partial | hold last valid partial or reset safely | stream sequence and texts in explicit debug | +| Backend timeout | clear provisional state and show recoverable status if appropriate | backend latency and timeout classification | +| Authentication failure | stop retries, prompt settings repair | endpoint identity and HTTP classification | +| Overlay geometry invalid | suppress or use deliberate fallback UI | geometry source/quality | +| Acceptance session stale | pass the original key through | accept preflight reason | +| IME active | use verified commit strategy | input source and insertion strategy | +| Paste menu unavailable | safe fallback or fail open | menu lookup/press result | +| Host write not published | bounded refresh, invalidate speculation/session | insertion attempt and content mismatch | +| Permission revoked | tear down consuming taps and surface guidance | permission transition | +| App sleeps/wakes | refresh permissions/focus/backend and discard old sessions | lifecycle generation | +| Process termination | stop new work, cancel/flush, release resources | shutdown stage durations | + +## Test Strategy + +### Pure unit tests + +Test value-based rules without a desktop: + +- Capability/availability decisions +- Request bounds and context budgets +- Work identity +- Session transitions +- Type-through +- Word/phrase segmentation +- Stream monotonicity +- Output normalization +- Geometry-mode policy +- Insertion planning +- Clipboard restore decisions +- Retry/timeout classification + +### Coordinator tests with fakes + +Inject fake focus, engine, overlay, and inserter boundaries. Test: + +- Late result rejection +- Field switch during generation +- Partial then final suppression +- Accept preflight failure passing through +- Host publication lag +- Permission and settings changes +- Backend switching + +### Controlled host fixtures + +Build small test hosts that expose known behavior: + +- Native NSTextField and NSTextView +- Secure and read-only fields +- Multiline and mid-line content +- WebKit contenteditable +- A deliberately delayed AX publisher if feasible +- An IME/manual composition procedure + +These fixtures make regressions reproducible before testing third-party applications. + +### Supported-application matrix + +For each declared host, test: + +- Empty, beginning, middle, and end-of-line caret +- Selection and replacement +- Rapid typing and deletion +- Partial and full acceptance +- Focus switch while streaming +- Undo/redo +- Multiple windows +- Multiple displays/Spaces/full screen +- Light/dark and accessibility display settings +- Relevant IMEs and RTL if in scope +- Permission revoke/regrant +- Network loss/recovery if hosted + +### Adversarial tests + +- Responses intentionally complete out of order +- Cancellation arrives during every async stage +- User changes clipboard during paste restore +- Acceptance is pressed twice rapidly +- Host exits during capture or insertion +- Selected engine/account changes during stream +- Machine sleeps during request +- AX returns malformed or non-finite geometry +- Backend sends control tokens, prompt echoes, or an empty final + +### Soak and resource tests + +- Continuous typing and focus switching +- Repeated overlay show/hide +- Repeated backend reconnect/prewarm +- Memory stability +- Event-tap recovery +- Idle CPU/wake behavior +- Clean termination under in-flight work + +## Candidate Alpha Gates + +These are starting proposals to calibrate with Josh, not promises before a baseline. + +### Non-negotiable correctness + +- Zero swallowed non-owned keys in automated classification/replay tests +- Zero stale results applied in the adversarial focus/request suite +- Zero focus-stealing overlay events +- Zero secure-field generation requests +- Zero clipboard loss in restore and overlapping-paste tests + +### Supported-host reliability + +- A measured insertion success target, such as at least 99 percent, in the declared application + matrix +- Every failure is either a known explicit degradation or produces a correlated diagnostic trail +- No silent accept where UI claims success but the host did not mutate + +### Performance + +- Define p50 and p95 time-to-first-useful-partial after measuring the current prototype +- Define p95 input-to-overlay update separately from backend latency +- No unbounded memory growth during repeated sessions +- Idle focus monitoring remains within an agreed CPU/wake budget + +### Stability and recovery + +- No crashes or hangs in the agreed soak scenario +- Permission revoke/regrant produces a recoverable state +- Network loss, authentication failure, and timeout do not leave a stale accept-ready session +- Sleep/wake and app relaunch clear incompatible state + +### Privacy + +- Context inventory matches product disclosure +- Secure-field behavior is covered by acquisition and request tests +- Credentials use Keychain or an equivalent platform secret boundary +- Full-content diagnostics are explicit, local/authorized, and retained according to policy + +## Scope Control + +A credible alpha scope is stronger than a universal claim. + +Define: + +- Supported macOS versions +- Supported processor requirements +- Named first-class applications +- Best-effort applications +- Unsupported sensitive or unusual fields +- Supported languages and IMEs +- Supported editing shapes +- One primary generation path +- One acceptance interaction +- Explicit offline/network behavior + +Potential early non-goals: + +- Every custom editor on macOS +- Perfect inline placement when caret geometry is unavailable +- Multiple generation backends +- Screenshot context +- Complex partial-word/phrase customization +- Broad application-specific settings +- Deterministic correction, emoji, or macros + +Non-goals are not admissions of failure. They protect the reliability promise. + +## Trial Deliverables to Discuss + +A strong working trial can produce: + +- A verified architecture map of the existing HyperWrite prototype +- Correlated diagnostics for the core suggestion loop +- An agreed alpha compatibility matrix +- A written reliability contract and measurable gates +- Hardened focus/input/session/insertion boundaries +- A tested primary generation path +- Permission, signing, and distribution readiness +- Known limitations and a post-alpha roadmap + +The exact selection depends on prototype maturity. Do not commit to all deliverables before seeing +the code and current build/release state. + +## How to Answer “What Would You Do First?” + +### Short answer + +> I would get the prototype running, reproduce one successful flow and the highest-impact failures, +> and add correlated stage logging if it is missing. Then I would write down the focus, work, +> session, input-consumption, and insertion invariants. That tells us whether we need targeted +> hardening or a deeper boundary change. I would prioritize wrong-target, swallowed-key, and +> insertion failures before model quality or feature breadth. + +### Deeper answer + +> My first deliverable would be a measurable baseline and risk map, not a rewrite. I would trace one +> suggestion from the focused editor through input, request, stream, overlay, acceptance, and host +> verification. I would make each stage share a request identity and classify current failures. Then +> I would harden the smallest trusted loop for an agreed app matrix: authoritative focus snapshot, +> fail-open input, cancellation plus work identity, one active interaction session, one generation +> contract, non-activating presentation, and verified insertion. Once that loop is reliable, we can +> broaden compatibility and context without multiplying unknowns. + +## How to Answer “Why Are You a Fit?” + +> Cotabby forced me to solve the same class of problems that turn a Mac autocomplete demo into a +> product: Accessibility inconsistency, browser and Electron behavior, global input without stealing +> focus, stale async generation, streaming session state, caret geometry, IMEs, safe insertion, +> permissions, native runtime lifecycle, and correlated debugging. I would not assume HyperWrite +> needs Cotabby's exact implementation, but I know which invariants to look for, how to isolate the +> risky boundaries, and how to scope reliability around evidence from real host applications. + +## Warning Signs During Scoping + +Ask for clarification if: + +- “Works everywhere” has no application matrix +- The acceptance key is consumed before insertion success is known +- Responses have no request/focus identity +- Raw editor or screenshot context is unbounded +- The prototype has no way to correlate a user report to one suggestion +- A network retry can outlive the editor session +- Overlay visibility is treated as the source of session truth +- Synthetic input is not distinguishable from physical input +- Hosted context transport is described as fully local +- Signing/TCC/update work is deferred until after the alpha + +Do not use warning signs to criticize the prototype. Use them to ask precise questions and propose a +reliability boundary. + +## Cotabby Study References for the Session + +Review these immediately before the discussion: + +1. [Root architecture map](../../ARCHITECTURE.md) +2. [Suggestion Pipeline](../architecture/suggestion-pipeline.md) +3. [Focus and Accessibility](../architecture/focus-and-accessibility.md) +4. [Input and Insertion](../architecture/input-and-insertion.md) +5. [Inference and Prompting](../architecture/inference-and-prompting.md) +6. [Context, Privacy, and Permissions](../architecture/context-privacy-and-permissions.md) +7. [Technical Decision Question Bank](technical-question-bank.md) + +The most relevant concrete source files are: + +- [CotabbyAppEnvironment.swift](../../Cotabby/App/Core/CotabbyAppEnvironment.swift) +- [FocusTracker.swift](../../Cotabby/Services/Focus/FocusTracker.swift) +- [FocusSnapshotResolver.swift](../../Cotabby/Services/Focus/FocusSnapshotResolver.swift) +- [InputMonitor.swift](../../Cotabby/Services/Input/InputMonitor.swift) +- [SuggestionWorkController.swift](../../Cotabby/Services/Suggestion/SuggestionWorkController.swift) +- [SuggestionInteractionState.swift](../../Cotabby/Services/Suggestion/SuggestionInteractionState.swift) +- [SuggestionCoordinator+Prediction.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator+Prediction.swift) +- [SuggestionCoordinator+Acceptance.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator+Acceptance.swift) +- [SuggestionInserter.swift](../../Cotabby/Services/Suggestion/SuggestionInserter.swift) +- [OverlayController.swift](../../Cotabby/Services/Presentation/OverlayController.swift) +- [SuggestionEngineRouter.swift](../../Cotabby/Services/Runtime/SuggestionEngineRouter.swift) +- [SuggestionRequestFactory.swift](../../Cotabby/Support/Suggestion/SuggestionRequestFactory.swift) +- [SuggestionSessionReconciler.swift](../../Cotabby/Support/Suggestion/SuggestionSessionReconciler.swift) + +## Final Position + +The interview is not about proving that Cotabby is perfect. It is about proving that you can: + +- Explain a complex existing system accurately +- Identify reliability invariants from first principles +- Name tradeoffs and current debt honestly +- Investigate a prototype before prescribing a rewrite +- Convert failure modes into boundaries, tests, and measurable alpha gates +- Scope product breadth around what users can trust + +That is the bridge from Cotabby architecture to HyperWrite product engineering. diff --git a/.internal/interview-prep/technical-question-bank.md b/.internal/interview-prep/technical-question-bank.md new file mode 100644 index 00000000..9d0f9180 --- /dev/null +++ b/.internal/interview-prep/technical-question-bank.md @@ -0,0 +1,877 @@ +# Cotabby Technical Decision Question Bank + +## How to Use This Bank + +These are not scripts to recite word for word. Learn the reasoning, then answer in your own voice. +Every strong answer should contain: + +1. The product or platform constraint +2. The chosen design +3. The failure it prevents +4. The cost or compromise +5. A concrete file or type +6. What you would improve with more time + +When Josh asks a short question, begin with the direct decision and stop after the tradeoff. Let him +pull you deeper. When he asks for a deep dive, use the source trail to walk from event to owner to +invariant. + +## Product and System Architecture + +### 1. What is the architecture of Cotabby? + +**Strong answer** + +Cotabby is a long-lived macOS menu bar agent organized around a cross-application autocomplete state +machine. It continuously reduces Accessibility state into a bounded FocusSnapshot, observes global +input without taking focus, builds an immutable request, routes it through one of three generation +backends, normalizes the output, materializes an active suggestion session, renders through a +non-activating AppKit panel, and validates insertion back into the host. + +The model is only one stage. Most reliability work is around eventual AX state, stale async work, +input ownership, geometry quality, and proving that an insertion actually reached another process. + +The dependency graph is constructed once by CotabbyAppEnvironment. AppDelegate controls startup and +shutdown. SuggestionCoordinator owns orchestration while pure rules live in Support and side effects +live in Services. + +**Source trail** + +- [CotabbyAppEnvironment.swift](../../Cotabby/App/Core/CotabbyAppEnvironment.swift) +- [AppDelegate.swift](../../Cotabby/App/Core/AppDelegate.swift) +- [SuggestionCoordinator.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator.swift) +- [Root architecture map](../../ARCHITECTURE.md) + +**Avoid** + +Do not answer, “It watches typing and calls llama.cpp.” That omits Apple, endpoints, insertion, and +the hard cross-process parts. + +### 2. What was the hardest technical part? + +**Strong answer** + +The hardest part is maintaining one coherent editing session across independent, eventually +consistent systems. CGEvents report the physical input before many applications publish their new +text through AX. AX elements can be replaced or recycled. Model work finishes asynchronously. The +overlay is in Cotabby's process while the caret is in another process. Synthetic insertion is a +request, not proof that the host mutated. + +The response was to make freshness explicit: work IDs, focusChangeSequence, bounded content +signatures, session anchors, overlay/session equality, post-insertion sentinels, and repeated +validation after awaits. + +**Source trail** + +- [SuggestionCoordinator+Input.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator+Input.swift) +- [SuggestionWorkController.swift](../../Cotabby/Services/Suggestion/SuggestionWorkController.swift) +- [SuggestionInteractionState.swift](../../Cotabby/Services/Suggestion/SuggestionInteractionState.swift) +- [SuggestionSessionReconciler.swift](../../Cotabby/Support/Suggestion/SuggestionSessionReconciler.swift) + +### 3. Why split App, Services, Models, Support, and UI? + +**Strong answer** + +The split follows change risk and side-effect ownership. Support holds deterministic policy, Models +hold shared values and contracts, Services own OS/native/network side effects, App coordinates them, +and UI renders state. That lets us test the acceptance rule or prompt budget without installing an +event tap or creating a panel. + +The tradeoff is more types and navigation. The benefit is that platform quirks do not become +unreviewable branches inside one coordinator. The split is useful only when each boundary owns a real +invariant; I would not create a file for every small function. + +**Source trail** + +- [SuggestionAvailabilityEvaluator.swift](../../Cotabby/Support/Suggestion/SuggestionAvailabilityEvaluator.swift) +- [SuggestionSubsystemContracts.swift](../../Cotabby/Models/Suggestion/SuggestionSubsystemContracts.swift) +- [SuggestionInserter.swift](../../Cotabby/Services/Suggestion/SuggestionInserter.swift) +- [SuggestionCoordinator.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator.swift) + +### 4. Is SuggestionCoordinator too large? + +**Strong answer** + +It is still a high-complexity owner because the product loop has many transitions, but it is no +longer intended as one monolithic algorithm. Its extensions separate lifecycle, input, prediction, +and acceptance. Mutable sub-state and pure decisions have moved into SuggestionWorkController, +SuggestionInteractionState, SuggestionAvailabilityEvaluator, SuggestionRequestFactory, and +SuggestionSessionReconciler. SuggestionStreamingState now owns partial coalescing/monotonic-render +bookkeeping, while PostExhaustionAcceptanceState owns the bounded rapid-Tab transition rules. + +I would not split the coordinator into independent actors merely to reduce file size because the +transitions need one MainActor ordering domain. I would continue extracting cohesive policies and +small state machines where they can be tested without duplicating ownership. + +**Source trail** + +- [SuggestionCoordinator.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator.swift) +- [SuggestionCoordinator+Lifecycle.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator+Lifecycle.swift) +- [SuggestionCoordinator+Input.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator+Input.swift) +- [SuggestionCoordinator+Prediction.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator+Prediction.swift) +- [SuggestionCoordinator+Acceptance.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator+Acceptance.swift) +- [SuggestionStreamingState.swift](../../Cotabby/Support/Suggestion/SuggestionStreamingState.swift) +- [PostExhaustionAcceptanceState.swift](../../Cotabby/Support/Suggestion/PostExhaustionAcceptanceState.swift) + +### 4A. Why introduce domain settings without migrating persistence? + +**Strong answer** + +The settings model accumulated fields from several product areas, but SwiftUI bindings, tests, and +UserDefaults keys already formed a compatibility surface. Replacing all of that at once would mix an +ownership improvement with a risky persistence migration. + +SuggestionSettingsData groups values into general, engine, completion, context, correction, +presentation, inline-feature, and shortcut domains. SuggestionSettingsModel projects its existing +published properties into that value, snapshots derive behavior from it, and SuggestionSettingsStore +continues mapping to the established flat keys. That improves the mental model without creating two +mutable sources or invalidating saved preferences. + +The tradeoff is a temporary forwarding layer. I would migrate consumers domain by domain only where +the cohesive value improves ownership, then remove compatibility accessors when call-site evidence +says it is safe. + +**Source trail** + +- [SuggestionSettingsData.swift](../../Cotabby/Models/Settings/SuggestionSettingsData.swift) +- [SuggestionSettingsModel.swift](../../Cotabby/Models/Settings/SuggestionSettingsModel.swift) +- [SuggestionSettingsStore.swift](../../Cotabby/Support/Settings/SuggestionSettingsStore.swift) +- [SuggestionSettingsDomainTests.swift](../../CotabbyTests/Models/Settings/SuggestionSettingsDomainTests.swift) + +## Ownership and Lifecycle + +### 5. Why is there one long-lived dependency graph? + +**Strong answer** + +Cotabby owns process-wide resources: Accessibility polling, event taps, runtime memory, settings, +panels, downloads, and permission state. If a SwiftUI redraw created a second FocusTracker or +InputMonitor, we could poll twice, consume a key twice, race model reloads, or display state from a +different settings instance. + +CotabbyAppEnvironment constructs those objects once and passes narrow collaborators into +coordinators. AppDelegate starts and stops side effects at process lifecycle boundaries. The cost is +a large composition root, but that cost is visible and deterministic. + +**Source trail** + +- [CotabbyApp.swift](../../Cotabby/App/Core/CotabbyApp.swift) +- [CotabbyAppEnvironment.swift](../../Cotabby/App/Core/CotabbyAppEnvironment.swift) +- [AppDelegate.swift](../../Cotabby/App/Core/AppDelegate.swift) + +### 6. Why do AppDelegate and CotabbyAppEnvironment both retain subscriptions? + +**Strong answer** + +They own different relationships. The environment retains relationships among the objects it +constructed, such as settings changing focus cadence, power profiles selecting engines, or endpoint +identity invalidating connection state. AppDelegate retains reactions tied to process lifecycle, +such as permissions refreshing input monitoring, engine changes loading or releasing llama, and +focus changes moving activation/debug overlays. + +The distinction is ownership, not “all subscriptions belong in one file.” If the environment did not +retain its cancellables for the process lifetime, graph-internal behavior would silently stop. + +**Source trail** + +- [CotabbyAppEnvironment.swift](../../Cotabby/App/Core/CotabbyAppEnvironment.swift) +- [AppDelegate.swift](../../Cotabby/App/Core/AppDelegate.swift) +- [Lifecycle and Composition](../architecture/lifecycle-and-composition.md) + +### 7. Why is native runtime shutdown synchronous at termination? + +**Strong answer** + +The llama context and Metal/native resources outlive ordinary Swift objects and can collide with C++ +static teardown if the process exits while work is active. AppDelegate first stops new coordination +and global input, then asks LlamaRuntimeManager for a bounded synchronous shutdown. LlamaRuntimeCore +prevents new work, aborts or waits for active operations under its lifecycle condition, and releases +native state. + +The tradeoff is that termination can wait briefly. The wait must be bounded because permission flows +sometimes require a prompt quit and relaunch. + +**Source trail** + +- [AppDelegate.swift](../../Cotabby/App/Core/AppDelegate.swift) +- [LlamaRuntimeManager.swift](../../Cotabby/Services/Runtime/LlamaRuntimeManager.swift) +- [LlamaRuntimeCore.swift](../../Cotabby/Services/Runtime/LlamaRuntimeCore.swift) + +## Accessibility and Editor Compatibility + +### 8. Why poll Accessibility instead of using AXObserver notifications? + +**Strong answer** + +AX notifications are incomplete and inconsistent across AppKit, browsers, Electron, and custom +editors. Mixing notification and polling streams also creates an ordering problem: which observation +is authoritative when they disagree? + +Cotabby uses one rule: a full capture is truth for that moment, and later captures repair stale +state. Activity can request refreshNow, but that still performs a capture rather than trusting the +event payload. Adaptive backoff reduces the idle cost. + +The tradeoff is synchronous AX IPC and periodic wakeups, so deep walks are bounded, throttled, cached, +and skipped when Cotabby is disabled. + +**Source trail** + +- [FocusTracker.swift](../../Cotabby/Services/Focus/FocusTracker.swift) +- [FocusPollBackoff.swift](../../Cotabby/Support/Focus/FocusPollBackoff.swift) +- [Focus and Accessibility](../architecture/focus-and-accessibility.md) + +### 9. How do you support Chrome and Electron? + +**Strong answer** + +Browser editors often expose the focused text node only after web accessibility is primed, and +out-of-process iframes can make the system focused-element query point at the wrong process or miss +the editor. Cotabby primes Chromium accessibility, resolves the actual owning application, and has a +cursor hit-test fallback that is cached only while it remains focused and belongs to the same browser +context. + +For geometry, browsers may expose text-marker bounds rather than reliable NSRange bounds. The +resolver tries marker geometry and static text runs before field estimation. Every fallback is +revalidated so it cannot mask a real focus change. + +**Source trail** + +- [FocusTracker.swift](../../Cotabby/Services/Focus/FocusTracker.swift) +- [ChromiumAccessibilityEnabler.swift](../../Cotabby/Services/Focus/ChromiumAccessibilityEnabler.swift) +- [FocusSnapshotResolver.swift](../../Cotabby/Services/Focus/FocusSnapshotResolver.swift) +- [AXTextGeometryResolver.swift](../../Cotabby/Services/Focus/AXTextGeometryResolver.swift) + +### 10. Why classify caret geometry quality? + +**Strong answer** + +An AX rectangle is not automatically an exact caret. Direct range bounds and derived nearby-character +bounds are precise enough for inline glyphs. A field-frame estimate may only identify the text line, +and hidden TextKit layout is still an estimate of a foreign editor. + +Cotabby carries exact, derived, estimated, or layoutEstimated quality into presentation. +CompletionRenderModePolicy uses inline for exact/derived and a mirror card for estimates or a +mid-line caret. That makes uncertainty visible in the UI instead of painting text over host content. + +The tradeoff is two presentation modes and more layout policy. It is preferable to confidently wrong +inline placement. + +**Source trail** + +- [FocusModels.swift](../../Cotabby/Models/Focus/FocusModels.swift) +- [AXTextGeometryResolver.swift](../../Cotabby/Services/Focus/AXTextGeometryResolver.swift) +- [CompletionRenderModePolicy.swift](../../Cotabby/Support/Presentation/CompletionRenderModePolicy.swift) +- [OverlayController.swift](../../Cotabby/Services/Presentation/OverlayController.swift) + +### 11. Why keep AX work on MainActor if it can be slow? + +**Strong answer** + +AX element access, AppKit state, focus caches, and publication are tightly coupled to main-thread +state. MainActor gives capture and cache mutation one ordering domain and avoids unsafe concurrent use +of Core Foundation/AX objects. + +The design does not pretend this is free. It bounds text, gates parameterized calls, caches +focus-session invariants, throttles deep descendants/static runs, and uses freshness checks to avoid +duplicate captures. OCR and model generation leave MainActor because they do not need live AX +objects. + +If profiling identified a specific safe extraction that could move off actor, I would introduce a +value-type boundary first rather than pass AXUIElement into arbitrary tasks. + +**Source trail** + +- [FocusTracker.swift](../../Cotabby/Services/Focus/FocusTracker.swift) +- [FocusSnapshotResolver.swift](../../Cotabby/Services/Focus/FocusSnapshotResolver.swift) +- [AXHelper.swift](../../Cotabby/Support/Accessibility/AXHelper.swift) + +## Async State and Reliability + +### 12. Why are cancellation and work IDs both necessary? + +**Strong answer** + +Task cancellation expresses intent, but an API or native decode can finish after cancellation was +requested. If result application only checked Task.isCancelled, a previous field's completion could +still appear. + +SuggestionWorkController increments a work ID whenever debounce/generation is replaced. A result can +apply only if its captured ID remains current. The coordinator also checks focus, content, settings, +and session signatures because a current task can still be invalidated by environment changes. + +The small cost is carrying identity through async boundaries. It converts a timing assumption into an +explicit invariant. + +**Source trail** + +- [SuggestionWorkController.swift](../../Cotabby/Services/Suggestion/SuggestionWorkController.swift) +- [SuggestionCoordinator+Prediction.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator+Prediction.swift) + +### 13. How do you handle a key event arriving before AX publishes the new value? + +**Strong answer** + +The listen-only CGEvent observes the key before many hosts update AXValue. Cotabby schedules after a +short debounce, requests a fresh capture, and performs bounded host-publication polling when it knows +the host is catching up. It also tracks capture freshness to avoid paying for redundant AX walks. + +There is a ceiling. Cotabby cannot wait forever for a broken host, so it eventually proceeds through +ordinary request and stale-result guards. This is one reason the pipeline is designed around eventual +consistency rather than assuming event order. + +**Source trail** + +- [SuggestionCoordinator+Input.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator+Input.swift) +- [FocusTracker.swift](../../Cotabby/Services/Focus/FocusTracker.swift) + +### 14. Why do you revalidate at several pipeline stages? + +**Strong answer** + +Eligibility is not a lease. Passing it before debounce does not authorize work after the user changes +fields, selects text, disables an app, switches engines, or accepts part of another session. + +Cotabby gates before scheduling to avoid waste, again before request construction, and again before +partial/final application and insertion. Each boundary checks the facts that could have changed while +awaiting. + +The tradeoff is repeated-looking guard code. Consolidating all guards into one early check would be +shorter but incorrect. + +**Source trail** + +- [SuggestionAvailabilityEvaluator.swift](../../Cotabby/Support/Suggestion/SuggestionAvailabilityEvaluator.swift) +- [SuggestionCoordinator+Prediction.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator+Prediction.swift) +- [SuggestionCoordinator+Acceptance.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator+Acceptance.swift) + +### 15. Why have a separate active suggestion session? + +**Strong answer** + +A completion is not just a string. It has an anchor focus/content signature, original prefix, +trailing text, consumed character count, kind, and expected host state. The user can type through it, +accept one chunk, switch fields, undo, select text, or race AX publication. + +SuggestionInteractionState owns mutable session facts. SuggestionSessionReconciler contains pure +transition rules. This makes the lifecycle explicit and testable rather than comparing the current +overlay string opportunistically. + +**Source trail** + +- [ActiveSuggestionSession.swift](../../Cotabby/Models/Suggestion/ActiveSuggestionSession.swift) +- [SuggestionInteractionState.swift](../../Cotabby/Services/Suggestion/SuggestionInteractionState.swift) +- [SuggestionSessionReconciler.swift](../../Cotabby/Support/Suggestion/SuggestionSessionReconciler.swift) + +### 16. Why can a streamed partial be accepted before generation finishes? + +**Strong answer** + +Autocomplete is latency-sensitive. If a cumulative partial is normalized, passes seam checks, and +materializes into the same session model as a final result, the user can act on useful text while +decode continues. + +SuggestionStreamingState makes pending partials latest-wins, permits one scheduled drain at a time, +and applies the monotonic rendering rule. The coordinator owns the runloop scheduling, freshness +checks, session creation, and overlay effects. The final is still authoritative: it can replace or +suppress a partial. Acceptance or new typing cancels or supersedes remaining work through the normal +work-ID rules. + +The tradeoff is more complex session coordination, but it reduces perceived latency without creating +a second acceptance system. + +**Source trail** + +- [StreamedGhostTextPolicy.swift](../../Cotabby/Support/Suggestion/StreamedGhostTextPolicy.swift) +- [SuggestionStreamingState.swift](../../Cotabby/Support/Suggestion/SuggestionStreamingState.swift) +- [SuggestionCoordinator+Prediction.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator+Prediction.swift) +- [SuggestionInteractionState.swift](../../Cotabby/Services/Suggestion/SuggestionInteractionState.swift) + +## Input and Insertion + +### 17. How do you guarantee Cotabby does not steal user keys? + +**Strong answer** + +The steady event tap is listen-only. The only suggestion tap capable of returning nil is installed +while interception is needed, and it consumes a configured key only after the current coordinator +successfully accepts. If the overlay disappeared, the session is stale, permission changed, or +insertion validation fails, the event passes through. + +That is fail-open behavior: uncertainty favors the host application. The global toggle has a separate +tap because its ownership is independent of suggestion visibility. + +There is one tightly bounded exception to overlay visibility: after the final visible chunk is +accepted, Cotabby can briefly retain Tab ownership while the next continuation regenerates. +PostExhaustionAcceptanceState collapses rapid presses to one queued accept and uses a generation-keyed +timeout so a stale callback cannot release a newer window. Teardown or timeout always returns Tab to +the host. + +**Source trail** + +- [InputMonitor.swift](../../Cotabby/Services/Input/InputMonitor.swift) +- [SuggestionCoordinator+Acceptance.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator+Acceptance.swift) +- [PostExhaustionAcceptanceState.swift](../../Cotabby/Support/Suggestion/PostExhaustionAcceptanceState.swift) + +### 18. How do you insert text into arbitrary applications? + +**Strong answer** + +For the common short single-line case, Cotabby posts a Unicode CGEvent pair. It is app-agnostic and +does not touch the clipboard. A composing IME uses paste because a synthetic Unicode event can be +absorbed into composition. An optional strategy also pastes long or multiline chunks. + +The paste path snapshots all pasteboard representations, places the completion temporarily, tries the +target app's AX Edit > Paste menu item, falls back to Command-V, and restores the clipboard only if no +newer clipboard activity occurred. + +There is no perfect universal insertion API. The architecture makes strategy and failure explicit, +then reconciles the host's later AX state. + +**Source trail** + +- [SuggestionInserter.swift](../../Cotabby/Services/Suggestion/SuggestionInserter.swift) +- [InsertionStrategySelector.swift](../../Cotabby/Support/Suggestion/InsertionStrategySelector.swift) +- [KeyboardInputSourceMonitor.swift](../../Cotabby/Services/Input/KeyboardInputSourceMonitor.swift) + +### 19. Why does Chrome need an AX Paste menu action? + +**Strong answer** + +A synthetic Command-V posted from inside a global tap callback can be ignored in Chrome while the +physical acceptance key is still down. Pressing the application's real Paste menu item drives the +same command through its own command routing and is more reliable. Cotabby caches the menu item per +process but validates every AXPress result; failure falls back to Command-V. + +This is a targeted compatibility workaround behind SuggestionInserter rather than a browser branch in +the coordinator. + +**Source trail** + +- [SuggestionInserter.swift](../../Cotabby/Services/Suggestion/SuggestionInserter.swift) +- [AXHelper.swift](../../Cotabby/Support/Accessibility/AXHelper.swift) + +### 20. How do you prevent synthetic insertion from triggering Cotabby again? + +**Strong answer** + +SuggestionInserter tags every synthetic event with a Cotabby-specific source marker and registers a +bounded expected event burst in InputSuppressionController. InputMonitor checks the marker first and +uses the count as a compatibility fallback when event transformation loses properties. + +The count accumulates across rapid acceptances because event delivery is asynchronous. It also +expires, so it cannot become a broad period during which real user input is ignored. + +**Source trail** + +- [InputSuppressionController.swift](../../Cotabby/Services/Input/InputSuppressionController.swift) +- [InputMonitor.swift](../../Cotabby/Services/Input/InputMonitor.swift) +- [SuggestionInserter.swift](../../Cotabby/Services/Suggestion/SuggestionInserter.swift) + +## Inference and Prompting + +### 21. Why support three generation backends behind one router? + +**Strong answer** + +The product needs one suggestion contract while devices and user privacy/performance choices differ. +Apple Intelligence is integrated with the OS, the in-process llama path works with downloaded base +GGUFs, and the OpenAI-compatible path supports loopback Ollama, LAN, or public HTTPS servers. + +SuggestionEngineRouter centralizes selection, metrics, prewarm forwarding, reset, and the narrow Apple +unsupported-language fallback. Backend implementations retain their own prompt, stream, lifecycle, +and transport mechanics. + +The tradeoff is a larger test matrix. The benefit is that coordinator and session logic do not fork +per backend. + +**Source trail** + +- [SuggestionEngineRouter.swift](../../Cotabby/Services/Runtime/SuggestionEngineRouter.swift) +- [SuggestionSubsystemContracts.swift](../../Cotabby/Models/Suggestion/SuggestionSubsystemContracts.swift) + +### 22. Why does Apple use a different prompt renderer from llama? + +**Strong answer** + +The llama models are base completion models. They condition on a text sequence; wrapping them in an +instruction conversation wastes tokens and can produce scaffolding instead of a continuation. +BaseCompletionPromptRenderer budgets optional context and places the caret prefix last. + +Apple's Foundation Models API provides a first-class instructions channel. FoundationModelPromptRenderer +uses that channel for policy and keeps content in the prompt. The domain request is shared, but the +backend-appropriate representation differs. + +**Source trail** + +- [BaseCompletionPromptRenderer.swift](../../Cotabby/Support/Prompting/BaseCompletionPromptRenderer.swift) +- [FoundationModelPromptRenderer.swift](../../Cotabby/Support/Prompting/FoundationModelPromptRenderer.swift) +- [SuggestionRequestFactory.swift](../../Cotabby/Support/Suggestion/SuggestionRequestFactory.swift) + +### 23. Why are LlamaRuntimeManager and LlamaRuntimeCore separate? + +**Strong answer** + +LlamaRuntimeManager is MainActor and ObservableObject because Settings and app lifecycle need +published model/loading/error state. LlamaRuntimeCore owns mutable native pointers, tokenization, +prompt sequence reuse, decode, abort, and shutdown. Those operations must not block UI and must obey +native serialization rules. + +The core is a nonisolated class marked unchecked Sendable with explicit locks and a lifecycle +condition. That is intentional: Sendable does not make native pointers thread-safe; the implementation +must enforce the synchronization. + +The split lets the manager express product state while the core protects native correctness. + +**Source trail** + +- [LlamaRuntimeManager.swift](../../Cotabby/Services/Runtime/LlamaRuntimeManager.swift) +- [LlamaRuntimeCore.swift](../../Cotabby/Services/Runtime/LlamaRuntimeCore.swift) + +### 24. Why not make LlamaRuntimeCore a Swift actor? + +**Strong answer** + +An actor would serialize Swift entry points, but the runtime also needs synchronous abort/shutdown +coordination, condition waiting around active native operations, and a narrow lock specifically +around prompt-cache/decode state. Native callbacks and C++ lifetime do not automatically become safe +because their wrapper is an actor. + +The explicit locks make the required critical sections and shutdown protocol visible. The tradeoff is +manual synchronization and unchecked Sendable responsibility. An actor could be a valid redesign, +but only if it preserved abort and bounded shutdown without blocking an executor or leaking pointers +across isolation. + +**Source trail** + +- [LlamaRuntimeCore.swift](../../Cotabby/Services/Runtime/LlamaRuntimeCore.swift) +- [LlamaRuntimeManager.swift](../../Cotabby/Services/Runtime/LlamaRuntimeManager.swift) + +### 25. How does prompt/KV-cache reuse remain safe? + +**Strong answer** + +The core tokenizes the new prompt, finds the longest safe reusable prefix against the prepared +sequence, reuses compatible evaluated tokens, and prefills the remainder. The autocomplete lock +serializes that cache state. Field/session changes broadcast reset, model changes rebuild native +state, and incompatible prompts fall back to a fresh sequence. + +The key rule is that cache reuse is an optimization under explicit continuity, never hidden +cross-field memory. If continuity is uncertain, correctness wins over latency. + +**Source trail** + +- [LlamaRuntimeCore.swift](../../Cotabby/Services/Runtime/LlamaRuntimeCore.swift) +- [LlamaSuggestionEngine.swift](../../Cotabby/Services/Runtime/LlamaSuggestionEngine.swift) +- [SuggestionCoordinator+Prediction.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator+Prediction.swift) + +### 26. Why normalize output outside each engine? + +**Strong answer** + +Every backend can emit control tokens, echoed prompt material, reasoning scaffolding, whitespace +seams, duplicated trailing text, or empty/unsafe output. If each engine cleans independently, users +get different acceptance semantics and fixes drift. + +SuggestionTextNormalizer provides one backend-independent contract. Engine-specific adapters produce +raw text and metadata; normalization and seam guards decide what can become ghost text. + +The tradeoff is that the shared normalizer must not accidentally encode quirks so narrowly that it +damages another backend. Detailed suppression reasons and tests help keep that visible. + +**Source trail** + +- [SuggestionTextNormalizer.swift](../../Cotabby/Support/Suggestion/SuggestionTextNormalizer.swift) +- [CompletionSeamGuard.swift](../../Cotabby/Support/Suggestion/CompletionSeamGuard.swift) +- [LlamaSuggestionEngine.swift](../../Cotabby/Services/Runtime/LlamaSuggestionEngine.swift) +- [FoundationModelSuggestionEngine.swift](../../Cotabby/Services/Runtime/FoundationModelSuggestionEngine.swift) + +## Context and Privacy + +### 27. What data does Cotabby use, and when can it leave the Mac? + +**Strong answer** + +A request can contain bounded text before and after the caret, app/window/domain/placeholder surface +metadata, user rules and extended context, a relevance-filtered clipboard excerpt, and cleaned +screenshot OCR when enabled and permitted. + +Apple Intelligence and the in-process llama engine generate on the Mac. The endpoint engine sends the +bounded constructed request to the configured server. Loopback stays on the machine, LAN leaves the +process/machine boundary to a local server, and public HTTPS leaves the local network. Credentials are +stored in Keychain, and insecure public HTTP is rejected. + +Debug mode is a separate privacy boundary because it can write full prompts, completions, AX dumps, +and screenshot/OCR artifacts locally. + +**Source trail** + +- [SuggestionRequestFactory.swift](../../Cotabby/Support/Suggestion/SuggestionRequestFactory.swift) +- [OpenAICompatibleEndpointModels.swift](../../Cotabby/Models/Runtime/OpenAICompatibleEndpointModels.swift) +- [CotabbyDebugOptions.swift](../../Cotabby/Support/Logging/CotabbyDebugOptions.swift) +- [Context, Privacy, and Permissions](../architecture/context-privacy-and-permissions.md) + +### 28. How does visual context work, and why not summarize OCR with another model? + +**Strong answer** + +VisualContextCoordinator owns one field-scoped session. WindowScreenshotService captures a compact +ScreenCaptureKit region, ScreenTextExtractor runs Vision OCR with per-line confidence, OCRTextHygiene +removes corruption and UI chrome, and ScreenshotContextGenerator sanitizes and caps the excerpt. + +There is no model summarization stage. A second generation adds latency, can hallucinate, complicates +privacy, and can destroy literal context a base completion model could condition on directly. The +tradeoff is that deterministic hygiene must handle OCR noise well. + +**Source trail** + +- [VisualContextCoordinator.swift](../../Cotabby/Services/Visual/VisualContextCoordinator.swift) +- [WindowScreenshotService.swift](../../Cotabby/Services/Visual/WindowScreenshotService.swift) +- [ScreenTextExtractor.swift](../../Cotabby/Services/Visual/ScreenTextExtractor.swift) +- [OCRTextHygiene.swift](../../Cotabby/Support/Context/OCRTextHygiene.swift) +- [ScreenshotContextGenerator.swift](../../Cotabby/Services/Visual/ScreenshotContextGenerator.swift) + +### 29. Are secure fields never captured? + +**Strong answer** + +No. The accurate current guarantee is narrower: secure fields are marked blocked, so they cannot +schedule generation, display a suggestion, accept text, or open emoji/macro capture. Their context +cannot reach a generation engine. + +However, FocusSnapshotResolver currently constructs a bounded FocusedInputSnapshot before returning +blocked capability, and visual-capture eligibility deliberately ignores capability, including secure +fields. With Screen Recording and visual context enabled, screenshot/OCR can run; explicit debug mode +can persist that capture. + +I would call that privacy debt rather than defend it. To reach a true no-capture guarantee, I would +move secure detection ahead of value/context construction and make visual eligibility reject secure +capability, then add tests proving no AX text, screenshot, OCR, or debug artifact is produced. + +**Source trail** + +- [FocusSnapshotResolver.swift](../../Cotabby/Services/Focus/FocusSnapshotResolver.swift) +- [SuggestionAvailabilityEvaluator.swift](../../Cotabby/Support/Suggestion/SuggestionAvailabilityEvaluator.swift) +- [VisualContextCoordinator.swift](../../Cotabby/Services/Visual/VisualContextCoordinator.swift) + +**Why this answer matters** + +An honest precise limitation demonstrates more trustworthiness than repeating an outdated privacy +claim. + +## Presentation and Product Decisions + +### 30. Why use AppKit panels instead of pure SwiftUI? + +**Strong answer** + +Cotabby needs a borderless panel above another app, on every Space and full-screen environment, +without becoming key, accepting mouse events, entering the window cycle, or stealing editor focus. +Those are NSPanel and NSWindow behaviors. + +OverlayController owns the panel and hosts typed SwiftUI content for the ghost or mirror view. This +uses SwiftUI where it is strong and AppKit for window semantics. The tradeoff is bridging two UI +systems, but a pure SwiftUI scene does not provide the required cross-application panel control. + +**Source trail** + +- [OverlayController.swift](../../Cotabby/Services/Presentation/OverlayController.swift) +- [SuggestionOverlayPresenter.swift](../../Cotabby/Services/Suggestion/SuggestionOverlayPresenter.swift) + +### 31. Why have emoji and macros inside an autocomplete app? + +**Strong answer** + +They reuse the same cross-application focus and insertion infrastructure but are deterministic, +low-latency productivity features. They do not call the model. Pure trigger state machines decide +whether colon or slash capture is active, and InlineCommandCoordinator arbitrates the one consuming +input slot. + +Architecturally, they demonstrate why InputMonitor should expose semantic capture ownership rather +than hard-code suggestion logic. The risk is feature interference, so their sigils are disjoint, +capture is pinned to one focus sequence, and the suggestion coordinator stands down when either +feature owns the event. + +**Source trail** + +- [InlineCommandCoordinator.swift](../../Cotabby/App/Coordinators/InlineCommandCoordinator.swift) +- [EmojiPickerController.swift](../../Cotabby/App/Coordinators/EmojiPickerController.swift) +- [MacroController.swift](../../Cotabby/App/Coordinators/MacroController.swift) +- [EmojiTriggerStateMachine.swift](../../Cotabby/Support/Emoji/EmojiTriggerStateMachine.swift) +- [MacroTriggerStateMachine.swift](../../Cotabby/Support/Macros/MacroTriggerStateMachine.swift) + +## Testing, Critique, and Engineering Judgment + +### 32. How do you test a system that depends on global macOS behavior? + +**Strong answer** + +I separate deterministic policy from OS boundaries. Work identity, request construction, +normalization, session reconciliation, trigger machines, geometry policy, prompt rendering, and +insertion planning have unit-testable value inputs. Coordinators depend on narrow protocols and fakes. + +Real AX, event taps, host publication, IMEs, browser behavior, signing, and TCC still require +integration/manual compatibility testing. Unit tests reduce the state-space before that matrix; they +do not pretend to replace it. + +CI independently checks compilation, tests, SwiftLint, and XcodeGen synchronization. + +**Source trail** + +- [CotabbyTests](../../CotabbyTests) +- [SuggestionSubsystemContracts.swift](../../Cotabby/Models/Suggestion/SuggestionSubsystemContracts.swift) +- [tests workflow](../../.github/workflows/tests.yml) +- [XcodeGen workflow](../../.github/workflows/xcodegen.yml) + +### 33. What are the biggest pieces of technical debt? + +**Strong answer** + +I would name debt precisely: + +1. Secure-field acquisition is broader than the privacy promise we want. +2. SuggestionCoordinator remains complex even after extracting pure rules and state owners. +3. Accessibility compatibility contains necessary app-specific branches that need a documented + compatibility matrix and regression harness. +4. Unit coverage is strong, but a repeatable end-to-end host compatibility harness is still the next + step for proving AX, input, presentation, and insertion together. +5. Endpoint support expands the privacy and availability matrix beyond the original on-device story. + +I would not propose a rewrite. I would first fix privacy-boundary tests and documentation, strengthen +integration replay/compatibility coverage, and continue extracting cohesive policy from the +coordinator only when ownership becomes clearer. + +**Source trail** + +- [SuggestionAvailabilityEvaluator.swift](../../Cotabby/Support/Suggestion/SuggestionAvailabilityEvaluator.swift) +- [SuggestionCoordinator.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator.swift) +- [FocusSnapshotResolver.swift](../../Cotabby/Services/Focus/FocusSnapshotResolver.swift) +- [SuggestionStreamingState.swift](../../Cotabby/Support/Suggestion/SuggestionStreamingState.swift) +- [PostExhaustionAcceptanceState.swift](../../Cotabby/Support/Suggestion/PostExhaustionAcceptanceState.swift) + +### 34. What would you do differently if you started Cotabby again? + +**Strong answer** + +I would define the interaction session and reliability invariants earlier: focus identity, +request/work identity, overlay/session equality, fail-open input, and post-insertion verification. I +would also establish an application compatibility matrix and correlated structured logging before +adding many app-specific fallbacks. + +I would keep the same broad boundaries—one dependency graph, normalized focus model, pure rules, +backend router, native core, and AppKit panel ownership—but make privacy acquisition gates and test +seams first-class from the first version. + +The useful lesson is not that the current code is wrong; it is that the difficult state-machine +properties became visible through real host applications and should be explicit earlier in the next +product. + +### 35. How do you know Cotabby is reliable? + +**Strong answer** + +Reliability is not one metric. I would separate: + +- Input integrity: no non-owned key is swallowed +- Freshness: no result applies to the wrong focus or text +- Insertion correctness: accepted text matches the planned mutation +- Presentation correctness: visible text matches session state and is anchored safely +- Availability: permissions and selected runtime degrade explicitly +- Latency: time to first useful partial and final result +- Resource behavior: idle AX cost, memory, model switch cleanup, shutdown +- Privacy: context acquisition, persistence, and transport match disclosure + +Cotabby has explicit guards and structured request-correlated logs for these, plus unit tests around +pure invariants. The next maturity step is a formal compatibility matrix and repeatable end-to-end +host harness that reports these dimensions per application. + +**Source trail** + +- [RequestID.swift](../../Cotabby/Support/Logging/RequestID.swift) +- [SuggestionDebugLogger.swift](../../Cotabby/Services/Suggestion/SuggestionDebugLogger.swift) +- [SuggestionSessionReconciler.swift](../../Cotabby/Support/Suggestion/SuggestionSessionReconciler.swift) +- [InputMonitor.swift](../../Cotabby/Services/Input/InputMonitor.swift) + +## HyperWrite Scoping Questions + +### 36. How would you approach turning the HyperWrite Mac prototype into a reliable alpha? + +**Strong answer** + +I would begin by observing the current prototype and defining its core interaction contract before +rewriting anything. For each suggestion I need to know: + +- What focused editor state is considered authoritative? +- What makes a request current or stale? +- Which component owns the active suggestion? +- Which keys may be consumed, and under what success condition? +- How is accepted text inserted and verified? +- What are the supported applications and explicit degradation modes? +- What context leaves the machine? +- Which correlated events let us explain a failure? + +I would instrument that loop first, then harden focus/input/insertion and stale-work invariants, +because model quality cannot compensate for dropped keys or text inserted into the wrong field. I +would preserve working prototype components behind narrow interfaces and replace only boundaries +whose failure data justifies it. + +The alpha should have a declared compatibility matrix and measurable gates for input integrity, +stale-result rejection, insertion success, crash-free operation, latency, memory, and privacy—not a +claim that every macOS text field works. + +**Further preparation** + +- [HyperWrite Reliability Translation](hyperwrite-reliability-translation.md) + +### 37. Would you copy Cotabby's architecture into HyperWrite? + +**Strong answer** + +I would transfer invariants, not copy the repository. The reusable ideas are one ownership graph, +normalized focus state, explicit work identity, a real interaction session, fail-open input, +strategy-based insertion, backend-independent normalization, geometry-quality-aware presentation, +bounded context, and request-correlated observability. + +I would not automatically copy polling cadence, app-specific AX fallbacks, local llama lifecycle, +prompt format, settings complexity, or the current coordinator shape. Those depend on HyperWrite's +prototype, backend, product promise, and supported application set. + +The first technical session should determine which failure modes are actually present. + +### 38. What would you prioritize if the trial is constrained? + +**Strong answer** + +I would prioritize the smallest loop that users can trust: + +1. Reliable focused-editor snapshot in the agreed application set +2. Fail-open input handling +3. Stale-request cancellation and identity +4. One validated generation path +5. Overlay/session consistency +6. Verified insertion, including IME policy +7. Correlated observability and repeatable compatibility tests +8. Permission/onboarding and distributable signing + +I would defer broad app claims, multiple speculative backends, elaborate settings, and optional +context until the core loop has measured reliability. That sequencing creates an alpha we can learn +from rather than a wide demo with unexplained failures. + +## Rapid-Fire Follow-Ups + +Use these to test whether you understand the answers rather than memorizing them: + +- What exact state invalidates one suggestion? +- What is the narrowest possible stale-result check, and why is it insufficient alone? +- Which code can legally return nil from a CGEvent tap? +- What happens when the visible overlay and session tail disagree? +- Which cache is allowed to survive ordinary typing? +- Which cache must reset on a field switch? +- What is the difference between local process, local machine, LAN, and public endpoint privacy? +- Which synchronous operation is most likely to hurt typing latency? +- Which native object is least safe to move across isolation? +- What proves that an insertion succeeded? +- Which bug would request_id let you diagnose that a plain error string would not? +- Which current product statement would you correct before a public endpoint launch? + +If you cannot answer a follow-up with a file and an invariant, return to the corresponding mastery +area in [the study path](README.md). diff --git a/AGENTS.md b/AGENTS.md index 2effb230..4bf27857 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -71,6 +71,12 @@ Start here when you need to understand lifecycle: and wires cross-subsystem subscriptions. SwiftUI views should observe objects from that graph rather than creating services directly. +`SuggestionSettingsModel` remains the individually `@Published` UI-facing compatibility surface. +Its `domainSettings` projection groups the same durable values into general, engine, completion, +context, correction, presentation, inline-feature, and shortcut domains. `SuggestionSettingsStore` +keeps the existing flat UserDefaults keys stable, and `SuggestionSettingsSnapshot` is the immutable +behavior boundary consumed by the suggestion pipeline. + This ownership rule prevents duplicate Accessibility observers, duplicate input monitors, runtime reload races, and mismatched settings state. @@ -84,7 +90,8 @@ Read the coordinator in this order: 4. `Cotabby/App/Coordinators/SuggestionCoordinator+Prediction.swift` 5. `Cotabby/App/Coordinators/SuggestionCoordinator+Acceptance.swift` -The coordinator owns orchestration and user-facing state. It should not absorb every rule. Prefer: +The coordinator owns orchestration plus active suggestion and presentation state. It should not +absorb every rule or state transition. Prefer: - `SuggestionRequestFactory` for pure request construction - `SuggestionAvailabilityEvaluator` for pure gating decisions @@ -92,6 +99,8 @@ The coordinator owns orchestration and user-facing state. It should not absorb e - `SuggestionTextNormalizer` for backend-independent output cleanup - `SuggestionWorkController` for generation task identity/cancellation - `SuggestionInteractionState` for active suggestion session storage +- `SuggestionStreamingState` for latest-wins partial coalescing and monotonic rendering state +- `PostExhaustionAcceptanceState` for the bounded rapid-accept window after a tail is exhausted This split matters because autocomplete is a state machine. Pure rules are easier to test and reason about than coordinator mutations. @@ -104,7 +113,8 @@ Focus and geometry live in: - `FocusSnapshotResolver`: reduces raw AX elements into Cotabby-supported focus snapshots. - `AXTextGeometryResolver`: resolves caret and input geometry. - `AXHelper`: low-level Accessibility/Core Foundation helper calls. -- `FocusModels`: pure focus values, identities, capabilities, and debug inspection data. +- `FocusModels`: pure focus values, identities, capabilities, stale-result signatures, and the + lightweight `FocusPollingEvent` used by the developer overlay. Accessibility data is eventually consistent and app-specific. Browser editors, Electron apps, native AppKit fields, and secure fields expose different AX shapes. Preserve stale-result guards, @@ -130,14 +140,15 @@ from Accessibility and Input Monitoring. Runtime generation is split by responsibility: -- `SuggestionEngineRouter`: selects Apple Intelligence vs Open Source. +- `SuggestionEngineRouter`: selects Apple Intelligence, Open Source, or the configured + OpenAI-compatible endpoint and owns the narrow Apple-to-llama language fallback. - `FoundationModelSuggestionEngine`: Apple on-device generation path. - `LlamaSuggestionEngine`: request-to-prompt, llama result handling, and cache reset handoff. +- `OpenAICompatibleSuggestionEngine`: completion/chat transport, SSE streaming, and Ollama preload + behavior for the configured endpoint. - `LlamaRuntimeManager`: UI-facing runtime state, model selection, warmup, and lifecycle control. - `LlamaRuntimeCore`: explicitly serialized native boundary around mutable llama.cpp pointers, - prompt tokenization, KV-cache reuse, sampling, an optional deterministic constrained decoder - (`runConstrainedDecode`, gated behind the default-off `cotabbyConstrainedDecoderEnabled`), and - shutdown. + prompt tokenization, KV-cache reuse, built-in sampler delegation, cancellation, and shutdown. - `BaseCompletionPromptRenderer`: prompt construction for the Open Source path. The llama models are now *base* (non-instruct) GGUFs, so this renders a pure text continuation: no instruction preamble, custom rules and context fold into a short conditioning preface (a base model conditions on diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 87e25feb..f86522b7 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -112,9 +112,12 @@ power-profile application, engine/model selection, and endpoint connection inval owns permission reactions, engine runtime start/stop, overlays, model-directory refresh, and process-lifecycle behavior. Both retain subscriptions because both own different relationships. -[SuggestionSettingsModel.swift](Cotabby/Models/Settings/SuggestionSettingsModel.swift) is the published source -of app behavior. [SuggestionSettingsStore.swift](Cotabby/Support/Settings/SuggestionSettingsStore.swift) -persists non-secret preferences in UserDefaults. Endpoint credentials live in Keychain. +[SuggestionSettingsModel.swift](Cotabby/Models/Settings/SuggestionSettingsModel.swift) is the +individually published UI-facing source of app behavior. Its +[SuggestionSettingsData.swift](Cotabby/Models/Settings/SuggestionSettingsData.swift) projection groups +the same values by product domain without replacing the existing API. The immutable snapshot used by +the pipeline is derived from those domains. [SuggestionSettingsStore.swift](Cotabby/Support/Settings/SuggestionSettingsStore.swift) +keeps the established flat UserDefaults keys stable; endpoint credentials live in Keychain. ## Suggestion State Machine @@ -126,17 +129,21 @@ Read the coordinator in this order: 4. [SuggestionCoordinator+Prediction.swift](Cotabby/App/Coordinators/SuggestionCoordinator+Prediction.swift) 5. [SuggestionCoordinator+Acceptance.swift](Cotabby/App/Coordinators/SuggestionCoordinator+Acceptance.swift) -The coordinator owns orchestration and user-facing suggestion state. It delegates rules and mutable -sub-state to smaller boundaries: +The coordinator owns orchestration plus active suggestion and presentation state. It delegates rules +and cohesive mutable sub-state to smaller boundaries: - [SuggestionAvailabilityEvaluator.swift](Cotabby/Support/Suggestion/SuggestionAvailabilityEvaluator.swift): pure permission, settings, focus, and runtime gates. - [SuggestionRequestFactory.swift](Cotabby/Support/Suggestion/SuggestionRequestFactory.swift): pure bounded - request construction and prompt preview. + request construction and the selected backend's developer-debug prompt payload. - [SuggestionWorkController.swift](Cotabby/Services/Suggestion/SuggestionWorkController.swift): debounce/generation tasks and monotonically increasing work IDs. - [SuggestionInteractionState.swift](Cotabby/Services/Suggestion/SuggestionInteractionState.swift): active session, materialized context, consumed prefix, and known post-insertion AX lag. +- [SuggestionStreamingState.swift](Cotabby/Support/Suggestion/SuggestionStreamingState.swift): latest-wins + partial coalescing, one scheduled drain, and monotonic rendered-text state. +- [PostExhaustionAcceptanceState.swift](Cotabby/Support/Suggestion/PostExhaustionAcceptanceState.swift): + pure state for the bounded Tab-ownership window while an exhausted tail regenerates. - [SuggestionSessionReconciler.swift](Cotabby/Support/Suggestion/SuggestionSessionReconciler.swift): type-through, acceptance, and live-host reconciliation. - [SuggestionTextNormalizer.swift](Cotabby/Support/Suggestion/SuggestionTextNormalizer.swift): backend-independent @@ -146,13 +153,17 @@ A native correction path runs before model generation. NSSpellChecker and bundle can suppress completion while a likely typo is forming, offer a green atomic replacement, or apply an opt-in automatic fix after Space. -Engines can stream cumulative partials. The coordinator coalesces UI work, accepts only monotonic -extensions, and allows a displayed partial to become an active accept-ready session. The final result +Engines can stream cumulative partials. SuggestionStreamingState coalesces token-rate callbacks into +latest-wins UI work, accepts only monotonic extensions, and lets a displayed partial become an active +accept-ready session. The coordinator still owns scheduling and presentation. The final result remains authoritative and can replace or suppress provisional text. Normal sessions support exact type-through, word or phrase acceptance, full-tail acceptance, CJK-aware segmentation, punctuation handling, optional trailing space, and speculative generation after final -acceptance. Corrections commit atomically rather than exposing partial acceptance. +acceptance. When rapid Tab presses cross the exhausted-tail regeneration gap, +PostExhaustionAcceptanceState can queue at most one unseen accept and has a generation-keyed backstop +that returns Tab ownership to the host. Corrections commit atomically rather than exposing partial +acceptance. ## Focus and Accessibility @@ -381,6 +392,7 @@ policies, prompt utilities, normalization, and layout logic should receive focus | Backend selection or memory issue | SuggestionEngineRouter, AppDelegate, LlamaRuntimeManager | | Native decode/cache/shutdown issue | LlamaRuntimeCore, CotabbyInference boundary | | Partial/final output mismatch | streaming policy, SuggestionTextNormalizer, seam guards | +| Rapid Tab leaks to the host after exhausting a tail | PostExhaustionAcceptanceState, acceptance coordinator | | Acceptance key passes through or is stolen | InputMonitor, acceptance validation | | Wrong or repeated inserted text | SuggestionInserter, InputSuppressionController, reconciler | | Clipboard context is irrelevant | ClipboardRelevanceFilter, ClipboardContentDistiller | @@ -388,6 +400,7 @@ policies, prompt utilities, normalization, and layout logic should receive focus | Emoji or macro conflicts with suggestions | InlineCommandCoordinator, feature trigger machine | | Permission loop or lost grant | PermissionManager, PermissionGuidanceController, app identity | | Settings/onboarding window issue | SettingsCoordinator, WelcomeCoordinator | +| Settings persistence or domain mismatch | SuggestionSettingsModel, SuggestionSettingsData, SuggestionSettingsStore | When a change crosses several rows, keep ownership at these boundaries rather than teaching one coordinator to perform every step itself. From 8171cbfa625ad35e7b8837e6d76ef98337943480 Mon Sep 17 00:00:00 2001 From: Jacob Fu <141651335+FuJacob@users.noreply.github.com> Date: Fri, 17 Jul 2026 22:39:57 -0700 Subject: [PATCH 2/3] Keep internal documentation local --- .gitignore | 1 + .internal/README.md | 85 -- .../context-privacy-and-permissions.md | 252 ----- .../architecture/focus-and-accessibility.md | 232 ----- .../architecture/inference-and-prompting.md | 239 ----- .internal/architecture/input-and-insertion.md | 199 ---- .../architecture/lifecycle-and-composition.md | 195 ---- .../presentation-and-sibling-features.md | 241 ----- .internal/architecture/suggestion-pipeline.md | 265 ------ .internal/interview-prep/README.md | 802 ---------------- .../hyperwrite-reliability-translation.md | 878 ------------------ .../interview-prep/technical-question-bank.md | 877 ----------------- 12 files changed, 1 insertion(+), 4265 deletions(-) delete mode 100644 .internal/README.md delete mode 100644 .internal/architecture/context-privacy-and-permissions.md delete mode 100644 .internal/architecture/focus-and-accessibility.md delete mode 100644 .internal/architecture/inference-and-prompting.md delete mode 100644 .internal/architecture/input-and-insertion.md delete mode 100644 .internal/architecture/lifecycle-and-composition.md delete mode 100644 .internal/architecture/presentation-and-sibling-features.md delete mode 100644 .internal/architecture/suggestion-pipeline.md delete mode 100644 .internal/interview-prep/README.md delete mode 100644 .internal/interview-prep/hyperwrite-reliability-translation.md delete mode 100644 .internal/interview-prep/technical-question-bank.md diff --git a/.gitignore b/.gitignore index 9840988a..3e5b784f 100644 --- a/.gitignore +++ b/.gitignore @@ -42,3 +42,4 @@ posts.txt .rocketride/ .agents +.internal diff --git a/.internal/README.md b/.internal/README.md deleted file mode 100644 index 6534bbb7..00000000 --- a/.internal/README.md +++ /dev/null @@ -1,85 +0,0 @@ -# Cotabby Internal Documentation - -This directory holds detailed maintainer and onboarding notes that are intentionally more exhaustive -than the repository's root architecture map. Start with [ARCHITECTURE.md](../ARCHITECTURE.md) for the -ten-minute system tour, then use the guides below to follow one responsibility through its owners, -data flow, invariants, and failure modes. - -The .internal directory is committed maintainer documentation. It may discuss implementation debt and -interview preparation more candidly and deeply than public product documentation, but it must not -contain credentials, private user data, or claims that cannot be verified from the repository. The -root architecture map remains self-contained and does not require these guides to be useful. - -## Architecture Guides - -| Guide | Question it answers | -| --- | --- | -| [Lifecycle and Composition](architecture/lifecycle-and-composition.md) | Who constructs, starts, retains, and stops app-lifetime objects? | -| [Suggestion Pipeline](architecture/suggestion-pipeline.md) | How does one input event become a streamed, visible, and accepted suggestion? | -| [Focus and Accessibility](architecture/focus-and-accessibility.md) | How does Cotabby resolve a safe field, bounded text, identity, and caret geometry? | -| [Input and Insertion](architecture/input-and-insertion.md) | Which global events are observed or consumed, and how is text committed safely? | -| [Inference and Prompting](architecture/inference-and-prompting.md) | How do Apple, llama, and endpoint requests differ while sharing one output contract? | -| [Context, Privacy, and Permissions](architecture/context-privacy-and-permissions.md) | What data can be acquired, where is it bounded, and when can it leave the Mac? | -| [Presentation and Sibling Features](architecture/presentation-and-sibling-features.md) | How do overlays, emoji, macros, settings, and onboarding share presentation infrastructure? | - -## Interview Preparation - -The interview-prep layer reuses the architecture guides rather than restating them: - -| Guide | Purpose | -| --- | --- | -| [Untimed Study Path](interview-prep/README.md) | Master the repository through exact source files, symbols, active-recall checkpoints, and six complete execution traces. | -| [Technical Decision Question Bank](interview-prep/technical-question-bank.md) | Practice difficult architecture questions with strong answers, tradeoffs, source trails, and honest limitations. | -| [HyperWrite Reliability Translation](interview-prep/hyperwrite-reliability-translation.md) | Apply Cotabby's lessons to inspecting, scoping, and hardening the HyperWrite Mac prototype into a measurable alpha. | - -## Suggested Reading Routes - -For the product loop and interview-level architecture discussion: - -1. Root ARCHITECTURE.md -2. Lifecycle and Composition -3. Suggestion Pipeline -4. Inference and Prompting -5. Context, Privacy, and Permissions - -For a field compatibility or ghost-positioning problem: - -1. Focus and Accessibility -2. Presentation and Sibling Features -3. Suggestion Pipeline - -For acceptance, IME, clipboard, or lost-keystroke behavior: - -1. Input and Insertion -2. Suggestion Pipeline -3. Focus and Accessibility - -For a new generation backend or context source: - -1. Inference and Prompting -2. Context, Privacy, and Permissions -3. Suggestion Pipeline -4. Lifecycle and Composition - -For the HyperWrite technical session: - -1. Untimed Study Path -2. Technical Decision Question Bank -3. HyperWrite Reliability Translation - -## Accuracy Standard - -These documents describe the current code, including uncomfortable limitations. They do not turn a -planned feature, stale comment, or desired privacy property into a shipping claim. - -When architecture changes: - -1. Verify behavior from the concrete owners and tests, not only comments or older docs. -2. Update the detailed guide whose responsibility changed. -3. Update root ARCHITECTURE.md only when the ten-minute mental model or a major invariant changes. -4. Keep source links resolvable and name the actual owner rather than only a folder. -5. Record a current limitation explicitly when implementation and desired invariant differ. - -The most important known example is secure-field acquisition: current generation and insertion fail -closed, but early bounded AX context and optional visual capture can still occur. The privacy guide -documents the exact boundary; do not simplify it to a no-capture guarantee until the code changes. diff --git a/.internal/architecture/context-privacy-and-permissions.md b/.internal/architecture/context-privacy-and-permissions.md deleted file mode 100644 index 12703c6e..00000000 --- a/.internal/architecture/context-privacy-and-permissions.md +++ /dev/null @@ -1,252 +0,0 @@ -# Context, Privacy, and Permissions - -## Purpose - -This guide inventories what Cotabby can observe, how each context source is bounded before -generation, which macOS permissions authorize the work, and when data can leave the Mac. Privacy is -an architectural constraint rather than a marketing label: every new context source needs an owner, -a permission boundary, a lifetime, a prompt budget, and a truthful disclosure for the selected -engine. - -## Context Flow - -One request can combine several independently enabled sources: - -~~~text -focused field - -> bounded preceding and trailing AX text - -> app, window, domain, placeholder, and surface metadata - -> optional user-authored rules and extended context - -> optional relevant clipboard description - -> optional screenshot-derived OCR excerpt - -> language and prediction settings - -> sanitization and per-section budgets - -> immutable SuggestionRequest - -> selected generation engine -~~~ - -[SuggestionRequestFactory.swift](../../Cotabby/Support/Suggestion/SuggestionRequestFactory.swift) is the final -assembly boundary. It receives values already acquired by their owning services; it does not capture -the screen, read the pasteboard, query AX, or open credentials itself. - -## Required and Optional Permissions - -[PermissionManager.swift](../../Cotabby/Services/Permission/PermissionManager.swift) publishes three -TCC grant states: - -- Accessibility is required to discover the focused editable element, read bounded text and - selection state, resolve geometry, validate acceptance, and perform some insertion fallbacks. -- Input Monitoring is required to observe global keyboard input and intercept accepted keys. -- Screen Recording is optional and used only for screenshot-derived visual context. - -Core autocomplete requires Accessibility and Input Monitoring. Missing Screen Recording does not -disable text-only suggestions; visual context reports an explicit unavailable state and the rest of -the pipeline continues. - -PermissionManager refreshes when Cotabby becomes active and uses a short polling loop only while a -required grant is missing. Once the required set is granted, it avoids a permanent permission poll. - -[PermissionGuidanceController.swift](../../Cotabby/Services/Permission/PermissionGuidanceController.swift) -owns how the user is guided through System Settings. It first invokes the native request API so -macOS registers the current signed application identity, then opens or overlays the relevant privacy -pane. Views ask for guidance; they do not implement TCC flow logic. - -The production and development application identities are intentionally distinct. A permission -grant belongs to the code identity macOS sees, so changing target identity, signing, or install path -can legitimately require a new grant. - -## Secure and Unsupported Fields - -[SecureFieldDetector.swift](../../Cotabby/Support/Accessibility/SecureFieldDetector.swift) centralizes detection -signals across native and non-native AX surfaces. A detected secure field receives blocked capability, -which prevents prediction, prompt construction, presentation, acceptance, and inline emoji or macro -capture. Surface metadata and field-style probes are also skipped. - -The current acquisition boundary is less strict than the assistance boundary. FocusSnapshotResolver -reads and bounds the field value, constructs FocusedInputSnapshot, and then returns the blocked -capability with that context still attached. SuggestionAvailabilityEvaluator also calls visual -capture with capability checking disabled; its code explicitly treats secure fields like other -transient blocked states so screenshot/OCR can warm early. - -That visual excerpt cannot enter an engine request because prediction remains blocked. However, the -screenshot and OCR still occur when Screen Recording and visual context are enabled, and explicit -debug mode can write the screenshot/OCR pair to disk. Therefore the accurate current guarantee is -"secure fields are never assisted or sent to an engine," not "secure fields are never captured." -Moving the secure check ahead of AX value construction and visual eligibility would be the path to -the stronger privacy invariant. - -Read-only, incompatible, explicitly disabled, or terminal surfaces are handled by capability and -availability policy. Disabling an application also prevents expensive compatibility walks where -possible, reducing both observation and the chance of disturbing a fragile AX tree. - -## Focused Text - -[FocusSnapshotResolver.swift](../../Cotabby/Services/Focus/FocusSnapshotResolver.swift) bounds text on -both sides of the caret before publication. This prevents a large document from propagating through -Combine state, signatures, logs, or request construction. SuggestionRequestFactory applies a smaller -engine-specific prefix budget later. - -Trailing text is retained only as much as Cotabby needs to avoid duplicating content after the caret -and to validate that a suggestion still belongs at the same seam. It is not an invitation to copy an -entire document into the prompt. - -## Surface Context - -[SurfaceContextComposer.swift](../../Cotabby/Support/Context/SurfaceContextComposer.swift) turns optional -application, window, browser-domain, placeholder, role, and surface metadata into a short descriptive -section. [SurfaceContextCache.swift](../../Cotabby/Services/Focus/SurfaceContextCache.swift) avoids -re-querying focus-session invariants on every poll. - -Surface metadata helps distinguish an email reply, browser editor, chat field, or document surface -without injecting an AX tree dump. Missing attributes remain absent; Cotabby does not fabricate -semantic certainty from a generic role. - -## User Context and Rules - -Extended context and custom writing rules are explicitly authored or enabled by the user. They are -sanitized and character-budgeted as optional prompt sections. These sections condition output but do -not outrank the recent caret prefix when prompt space is scarce. - -Rules, languages, and settings are durable non-secret preferences stored through -SuggestionSettingsStore. SuggestionSettingsData groups those values by product domain in memory, -while the store preserves the established flat UserDefaults keys for compatibility. Endpoint bearer -tokens are different: they are secrets and live in Keychain. - -## Clipboard Context - -[ClipboardContextProvider.swift](../../Cotabby/Services/Context/ClipboardContextProvider.swift) -reads a fresh bounded description only when the coordinator is building context. It does not publish -or retain a continuous clipboard history. Text is normalized; images and image files become compact -metadata descriptions rather than raw pixel prompt payloads. - -[ClipboardRelevanceFilter.swift](../../Cotabby/Support/Context/ClipboardRelevanceFilter.swift) prevents an -unrelated pre-existing clipboard from entering every request. It tracks pasteboard identity and -recentness, requires meaningful token overlap with the current prefix for longer content, and pins an -accepted relevance verdict to the active field so normal typing does not make context flicker. - -[ClipboardContentDistiller.swift](../../Cotabby/Support/Context/ClipboardContentDistiller.swift) keeps short -content intact and extracts overlapping lines from longer content. If no useful line can be found, -it falls back to a bounded head rather than carrying an unbounded clipboard dump. - -Clipboard context and clipboard-based insertion are separate concerns. Disabling clipboard prompt -context does not disable the IME-safe insertion fallback, which touches the pasteboard transiently and -restores the user's representations. - -## Visual Context Lifecycle - -[VisualContextCoordinator.swift](../../Cotabby/Services/Visual/VisualContextCoordinator.swift) owns -one field-scoped visual session. On a supported focus it: - -1. Coalesces rapid focus churn so Chromium and Electron settling does not launch duplicate work. -2. Checks whether visual context is enabled and Screen Recording is granted; capability is currently - ignored here, including secure-field capability. -3. Creates a session ID tied to the focused input. -4. Runs screenshot and OCR work asynchronously. -5. Applies the excerpt only if the session is still current. -6. Keeps the result through ordinary typing in the same field. -7. Cancels and clears it when focus continuity ends. - -Status is explicit: disabled, waiting, capturing, extracting text, ready, unavailable, or failed. -Missing optional permission is therefore distinguishable from an OCR failure and from a feature the -user turned off. - -## Screenshot to OCR - -[WindowScreenshotService.swift](../../Cotabby/Services/Visual/WindowScreenshotService.swift) uses -ScreenCaptureKit to select the visible window belonging to the focused process and capture a compact -region around the input. It excludes desktop windows and converts coordinate systems at the capture -boundary. - -[ScreenTextExtractor.swift](../../Cotabby/Services/Visual/ScreenTextExtractor.swift) runs Apple Vision -OCR and carries recognition confidence per line. [OCRTextHygiene.swift](../../Cotabby/Support/Context/OCRTextHygiene.swift) -then removes low-confidence lines, replacement characters, symbol-heavy noise, likely UI chrome, -digit-corrupted prose, and lines that echo the field's own text. - -[ScreenshotContextGenerator.swift](../../Cotabby/Services/Visual/ScreenshotContextGenerator.swift) -normalizes, sanitizes, checks meaningful signal, and applies a final excerpt cap. It caches a few raw -OCR extractions by sampled pixel hash, but reruns hygiene and field-text stripping on each use. - -There is no model summarization step. The prompt receives cleaned, bounded OCR text. This avoids a -second generation, avoids summary hallucination, and lets a base completion model condition on the -visible language directly. Raw screenshots never enter the suggestion prompt. - -## Prompt Sanitization and Budgets - -[PromptContextSanitizer.swift](../../Cotabby/Support/Context/PromptContextSanitizer.swift) normalizes optional -context and caps it before rendering. BaseCompletionPromptRenderer applies per-section budgets again -when assembling the final continuation prompt. - -The two layers serve different purposes: acquisition boundaries prevent oversized values from -circulating through the app, while prompt budgets decide which already-safe sections fit in this -particular request. No context source should rely only on a last-minute total-string truncation. - -## Where Data Is Processed - -Apple Intelligence and the in-process llama engine perform suggestion generation on the Mac. The -visual pipeline also uses local ScreenCaptureKit and Vision APIs. - -The OpenAI-compatible engine sends the constructed request to the endpoint the user configured. A -loopback Ollama address stays on the Mac, a LAN address sends it across the local network, and a -public HTTPS address sends it to that remote service. Endpoint privacy classification and settings UI -must communicate that distinction. Insecure public HTTP is rejected. - -Model discovery/downloads and update checks can use the network, but those operations are separate -from sending focused text for generation. Adding any hosted generation dependency or telemetry path -requires an explicit product decision and documentation update. - -## Secrets and Persistence - -- Non-secret settings persist in UserDefaults through SuggestionSettingsStore. -- Endpoint bearer tokens persist in the user's Keychain. -- Focus snapshots, clipboard context, visual excerpts, and active requests are memory-scoped values. -- Visual OCR cache entries are small in-memory extractions bounded to a few pixel hashes. -- No normal product path maintains a document, clipboard, screenshot, or prompt history database. - -## Diagnostics Are a Privacy Mode - -Normal operation sends structured events to Apple's unified logging. Launching with --cotabby-debug intentionally enables privacy-sensitive local artifacts for development: - -- ~/Library/Logs/Cotabby/cotabby.jsonl for structured events -- ~/Library/Logs/Cotabby/llm-io.jsonl for full prompts and completions -- ~/Desktop/cotabby-ax-dump.txt for the latest Chrome AX tree snapshot -- ~/Desktop/cotabby-debug-screenshots/ for bounded retained screenshot/OCR pairs - -The LLM-I/O records share request IDs with the main stream. Visual debug captures are retained only -under the explicit debug gate and pruned per application. Anyone collecting a bug report must treat -these artifacts as potentially containing user text and screen content. - -## Invariants - -- Secure fields never schedule generation, presentation, acceptance, or inline-command capture. -- Current AX and visual acquisition can still run before/around the secure capability block; this is - a documented privacy limitation, not a no-capture guarantee. -- Accessibility and Input Monitoring gate core autocomplete; Screen Recording remains optional. -- Every text source is bounded before request assembly. -- Optional sections are sanitized and budgeted independently. -- Clipboard context is relevance-filtered and is not a continuous history. -- A visual excerpt belongs to one field-scoped session and stale work cannot apply. -- Visual context uses cleaned OCR, never raw screenshots or model-generated summaries. -- On-device and configured-endpoint generation are disclosed as different privacy scopes. -- Endpoint secrets live in Keychain. -- Privacy-sensitive files exist only when the developer explicitly launches debug mode. - -## Failure-Oriented Reading - -- Suggestions run with a missing required grant: PermissionManager and availability evaluation. -- Password field receives a ghost, generation request, or command capture: secure detection and - focus capability. -- Requirement is that secure fields are never captured at all: current focus-value and visual-context - acquisition must be changed; the existing block applies later. -- Clipboard text appears unrelated: relevance identity, overlap, and field pinning. -- Visual context follows the previous field: session ID and focus-scoped apply guard. -- Screenshot context is noisy: OCR confidence, hygiene order, and prompt sanitizer. -- Screen Recording denial disables all suggestions: required-versus-optional permission policy. -- Public endpoint has no warning: endpoint scope classification and settings presentation. -- Prompt content appears on disk unexpectedly: CotabbyDebugOptions bootstrap and launch arguments. - -## Update This Guide When - -Update this document whenever Cotabby observes a new data source, changes a permission requirement, -changes secure-field policy, adds a persistent cache, changes endpoint behavior, writes a new debug -artifact, or alters a context budget or lifetime. diff --git a/.internal/architecture/focus-and-accessibility.md b/.internal/architecture/focus-and-accessibility.md deleted file mode 100644 index 28c6af04..00000000 --- a/.internal/architecture/focus-and-accessibility.md +++ /dev/null @@ -1,232 +0,0 @@ -# Focus and Accessibility - -## Purpose - -Cotabby must understand text and caret geometry inside applications it does not own. macOS -Accessibility is the only cross-application semantic interface available for that job, but its data -is synchronous, cross-process, app-specific, and eventually consistent. This subsystem converts raw -AX state into a bounded, capability-checked FocusSnapshot that the rest of Cotabby can consume. - -## Main Pipeline - -Read these files in order: - -1. [FocusTracker.swift](../../Cotabby/Services/Focus/FocusTracker.swift) -2. [FocusSnapshotResolver.swift](../../Cotabby/Services/Focus/FocusSnapshotResolver.swift) -3. [FocusModels.swift](../../Cotabby/Models/Focus/FocusModels.swift) -4. [AXTextGeometryResolver.swift](../../Cotabby/Services/Focus/AXTextGeometryResolver.swift) -5. [AXHelper.swift](../../Cotabby/Support/Accessibility/AXHelper.swift) - -The high-level flow is: - -~~~text -frontmost or owning application - -> focused AX element - -> candidate editable elements - -> capability and secure-field checks - -> bounded text and selection - -> caret and field geometry - -> stable focus identity and generation - -> FocusSnapshot publication -~~~ - -## Why Polling Is Authoritative - -FocusTracker uses polling as its single source of truth. AXObserver notifications are inconsistent -across native, browser, Electron, and custom editors. Mixing notification and poll streams would -also create ordering ambiguity. Polling provides one eventual-consistency rule: every capture -re-reads the current field, and stale state repairs on a later capture. - -Input and acceptance paths can request refreshNow when they know a fresh read is useful. Those are -still polling-style full captures, not trusted event payloads. FocusTrackingModel exposes the age of -the most recent capture so multiple pipeline stages can avoid paying for identical AX work within a -short freshness window. - -## Adaptive Polling - -Active editing uses the configured base cadence. When repeated captures produce no focus or content -change, FocusPollBackoff stretches the timer interval. Explicit activity resets the backoff. This -keeps editing responsive while avoiding continuous main-thread wakeups and deep AX walks on an idle -machine. - -The timer is re-created only when the effective interval changes. That prevents no-op timer churn on -every keypress. - -## Resolving the Owning Application - -The app that owns the focused AX element is more reliable than NSWorkspace.frontmostApplication. -Accessory utilities such as launchers can own a focused non-activating panel while another -application remains nominally frontmost. Per-app disable policy and surface classification must use -the actual focused process when possible. - -Chromium out-of-process frames complicate ownership because a focused AX node can belong to a -renderer subprocess. The Chromium fallback deliberately associates a hit-tested element with the -visible browser application. - -## Chromium and Electron - -Chromium and Electron surfaces require extra compatibility work: - -- ChromiumAccessibilityEnabler primes web accessibility so renderer text becomes visible. -- A cursor hit-test fallback can recover focused editors that the system-wide focused-element query - misses, including some out-of-process iframe editors. -- The hit-tested element is cached only while it still reports focus and the browser identity - remains valid. -- Browser detection and web-content classification affect candidate choice and geometry policy. -- Debug builds can write a focused Chrome AX tree snapshot for inspection. - -These fallbacks must never mask a real focus change. Every cached element is revalidated, and normal -system focus wins immediately when it becomes available. - -## Candidate Resolution and Capability - -FocusSnapshotResolver searches around the focused element for the most usable editable candidate. -It reduces AX implementation details into domain values: - -- Application and process identity -- Element identity and focus-change sequence -- Preceding and trailing text around the selection -- Selection location and length -- Field and caret rectangles -- Geometry quality -- Window, URL, placeholder, and surface metadata -- Capability and block reason - -FocusCapabilityResolver and related pure policies decide whether the result is supported. Secure -fields, unsupported roles, read-only surfaces, incompatible selections, terminals under default -policy, and explicitly disabled applications must fail safely at the assistance boundary. - -FocusSnapshot intentionally carries only application identity, capability, and optional bounded -context. The obsolete detailed inspection snapshot was removed; developer visibility now travels -through the lightweight FocusPollingEvent plus caret/field geometry and visual-context status. This -keeps normal focus values smaller without removing the debug overlay's polling evidence. - -There is a current acquisition caveat: the resolver constructs a bounded FocusedInputSnapshot before -returning blocked capability for a secure field, and the separate visual-capture gate ignores -capability. Suggestions and insertion remain blocked, but "blocked" does not currently mean that no -AX value or screenshot was acquired. Treat the context/privacy guide as authoritative for that -distinction. - -Cotabby normally blocks its own process. The Context settings pane has one sanctioned live-preview -field identified by a known AX identifier; every other Cotabby field remains excluded. - -## Bounded Text - -Focus snapshots do not carry entire documents. Text on each side of the caret is capped before it -flows through equality checks, Combine publication, stale-result signatures, or prompt construction. -The prompt factory applies a smaller engine-specific window later. - -Bounding at the AX boundary matters because a large editor can otherwise make every focus capture, -comparison, and request signature scale with document size. - -## Focus Identity - -No single AX identifier is perfectly stable. Chromium can recycle or replace AX nodes during normal -editing. Cotabby combines: - -- Process identity -- Element identity where useful -- A monotonic focusChangeSequence -- Selection and text signatures -- Generation/content signatures - -Different consumers use the narrowest identity appropriate to their invariant. Visual context needs -field-scoped uniqueness, while active-session reconciliation intentionally favors process and text -continuity over brittle AX node identity. - -## Geometry Resolution - -AXTextGeometryResolver chooses the best caret geometry available: - -1. Zero-length AXBoundsForRange at the caret -2. AX text-marker range geometry used by some browser engines -3. Bounds of the character before the caret, shifted to its trailing edge -4. Child static-text-run frames with proportional placement -5. Field-frame estimation as a last resort - -The result carries a CaretGeometryQuality: - -- exact: direct caret/range geometry -- derived: geometry inferred from a nearby measured character -- estimated: coarse field-based fallback -- layoutEstimated: repaired later using hidden TextKit layout - -Quality is part of presentation policy. Exact and derived geometry can support inline ghost text. -Estimated or layout-repaired positions normally use a Cotabby-owned mirror card because small -horizontal errors are visually obvious when glyphs are painted directly beside host text. - -## Geometry Cost Controls - -AX calls are synchronous IPC and can block the MainActor while the target process responds. -Compatibility walks therefore use several controls: - -- Gate parameterized attributes on advertised support. -- Validate returned rectangles against an anchor frame. -- Throttle deep descendant searches. -- Throttle static-text-run walks. -- Cache field style and surface metadata within a focus session. -- Invalidate transient caret caches after Cotabby mutates the field. -- Reuse a recent suggestion anchor when AX frames briefly regress after insertion. - -These optimizations are correctness-sensitive. A cache key must include the identity or generation -needed to prevent data from one field leaking into another. - -## App-Specific Safety - -Some applications react badly to AX enumeration. Calendar can dismiss a transient date/time editor -when its tree is walked. CalendarAccessibilityCaptureGuard uses pointer context to suppress only the -fragile interaction window instead of disabling the whole application. - -Per-app disable and global pause are checked before expensive candidate walks. This is both a -performance optimization and a compatibility guarantee: disabled Cotabby should not perturb the -target application's AX tree. - -## Coordinate Systems - -Accessibility, AppKit, Core Graphics, and Vision can describe screen rectangles using different -origins and coordinate conventions. AXHelper and DisplayCoordinateConverter centralize conversion. -Callers should not hand-roll Y-axis flips or mix AX and Cocoa rectangles without an explicit -conversion boundary. - -Every rectangle that reaches AppKit presentation is checked for finite components. A malformed AX -frame should suppress presentation, not crash NSPanel layout. - -## Concurrency and Freshness - -Most AX access is MainActor-isolated. This makes mutation of focus caches and publication ordered, -but it also means expensive synchronous AX calls can affect input responsiveness. The subsystem -therefore emphasizes bounding, throttling, freshness checks, and avoiding duplicate captures. - -Async consumers must assume the snapshot can become stale immediately after they read it. They carry -focus and content signatures through awaits and validate again before applying work. - -## Invariants - -- Polling is the authoritative focus source. -- Cached Chromium fallbacks never outrank a valid system-focused element. -- Secure and unsupported fields fail closed for prediction, presentation, and insertion; early - secure-field acquisition remains a documented privacy limitation. -- Captured text is bounded before publication. -- Deep AX work stops when Cotabby is disabled or interaction safety requires it. -- Geometry always carries a quality classification. -- AX rectangles are converted and validated at explicit boundaries. -- Async results never rely on AX element identity alone. -- Cotabby's own fields remain blocked except for the explicit live-preview field. - -## Failure-Oriented Reading - -- Field is not detected: FocusTracker focus acquisition, Chromium priming/hit test, candidate search. -- Wrong app gets per-app policy: owning-application resolution. -- Secure or read-only field is treated as editable: capability resolver and secure-field detection. -- Ghost appears at field edge: geometry fallback and quality classification. -- Ghost jitters after typing or acceptance: geometry caches, anchor stability, post-insertion - invalidation. -- Idle app consumes CPU: FocusPollBackoff and deep-walk throttles. -- Calendar popover closes: CalendarAccessibilityCaptureGuard and suppression boundary. -- Browser works until iframe focus: Chromium OOPIF hit-test cache and revalidation. - -## Update This Guide When - -Update this document when focus acquisition gains a new source, identity semantics change, a new -geometry branch or quality is added, AX context bounds change architecturally, or an app-specific -compatibility guard changes what Cotabby is allowed to inspect. diff --git a/.internal/architecture/inference-and-prompting.md b/.internal/architecture/inference-and-prompting.md deleted file mode 100644 index 8b9c59d3..00000000 --- a/.internal/architecture/inference-and-prompting.md +++ /dev/null @@ -1,239 +0,0 @@ -# Inference and Prompting - -## Purpose - -This guide explains how one structured SuggestionRequest becomes a completion through Apple -Intelligence, the in-process llama runtime, or an OpenAI-compatible endpoint. It also describes where -prompt construction, streaming, normalization, cache reuse, credentials, and native resource -ownership belong. - -The key boundary is that engines generate from an immutable request. They do not read live focus or -settings state while decoding. Backend-specific transport and prompting stay behind the shared -SuggestionGenerating contract, while backend-independent output cleanup stays in Support. - -## Main Files - -Start with: - -1. [SuggestionEngineRouter.swift](../../Cotabby/Services/Runtime/SuggestionEngineRouter.swift) -2. [SuggestionRequestFactory.swift](../../Cotabby/Support/Suggestion/SuggestionRequestFactory.swift) -3. [BaseCompletionPromptRenderer.swift](../../Cotabby/Support/Prompting/BaseCompletionPromptRenderer.swift) -4. [FoundationModelPromptRenderer.swift](../../Cotabby/Support/Prompting/FoundationModelPromptRenderer.swift) -5. [SuggestionTextNormalizer.swift](../../Cotabby/Support/Suggestion/SuggestionTextNormalizer.swift) - -Then follow the selected backend: - -- Apple: [FoundationModelSuggestionEngine.swift](../../Cotabby/Services/Runtime/FoundationModelSuggestionEngine.swift) -- In-process llama: [LlamaSuggestionEngine.swift](../../Cotabby/Services/Runtime/LlamaSuggestionEngine.swift), - [LlamaRuntimeManager.swift](../../Cotabby/Services/Runtime/LlamaRuntimeManager.swift), and - [LlamaRuntimeCore.swift](../../Cotabby/Services/Runtime/LlamaRuntimeCore.swift) -- Endpoint: [OpenAICompatibleSuggestionEngine.swift](../../Cotabby/Services/Runtime/OpenAICompatibleSuggestionEngine.swift), - [OpenAICompatibleAPIClient.swift](../../Cotabby/Services/Runtime/OpenAICompatibleAPIClient.swift), - and [OpenAICompatibleEndpointModels.swift](../../Cotabby/Models/Runtime/OpenAICompatibleEndpointModels.swift) - -## Routing - -SuggestionEngineRouter owns backend selection, not model lifecycle or endpoint transport. The -settings snapshot selects one of three paths: - -~~~text -Apple Intelligence - -> FoundationModelSuggestionEngine - -Open Source - -> LlamaSuggestionEngine -> LlamaRuntimeManager -> LlamaRuntimeCore -> CotabbyInference - -OpenAI-compatible endpoint - -> OpenAICompatibleSuggestionEngine -> OpenAICompatibleAPIClient -> configured server -~~~ - -Single-result generation and streaming are both forwarded to the selected backend. Prewarm is sent -only to the selected backend. A context reset is broadcast to all backends because an old continuation -cache must not survive a field or session boundary merely because that backend is temporarily -unselected. - -Apple generation falls back to the in-process llama engine only for unsupported locale or language -conditions. A normal Apple generation failure is surfaced rather than silently changing engines and -making behavior difficult to diagnose. - -## Request Construction - -SuggestionRequestFactory converts a verified focus snapshot and context bundle into an immutable -request. It chooses an engine-appropriate prefix bound, language-aware prediction budget, enabled -context sections, request identifier, and selected-backend prompt payload for explicit developer -debug logging. - -Optional context is sanitized and budgeted before it reaches a renderer. The engine receives both -structured fields and a base-completion prompt so Apple can use its first-class instructions channel -while llama and completion-style endpoints can consume ordinary continuation text. - -The caret prefix is the most important signal. Optional context can be reduced or omitted to preserve -the recent text immediately before the caret. - -## Two Prompt Shapes - -The open-source GGUF models are base completion models, not instruction-tuned chat models. -BaseCompletionPromptRenderer therefore builds a text continuation: - -- A short conditioning preface can describe language, style, or custom rules. -- Optional surface, clipboard, visual, and extended context are bounded by PromptSectionBudget. -- The actual caret prefix appears last so generation continues from the user's text. -- There is no instruction-conversation wrapper that asks the model to obey commands. - -The Apple Foundation Models API supplies a first-class instructions channel. FoundationModelPromptRenderer -keeps instruction-shaped policy separate from the user content instead of pretending the Apple path -is a raw base-model continuation. - -OpenAI-compatible endpoints support completion and chat request modes, but both are derived from the -same Cotabby request and its bounded prompt. Mode changes transport shape; they do not authorize the -endpoint engine to reacquire unbounded editor context. - -## Apple Intelligence - -FoundationModelSuggestionEngine owns Apple framework availability checks, language support, session -creation, prewarm, generation, and streaming. It uses LanguageModelSession.streamResponse for -cumulative partials and retains at most one prepared session for the next compatible request. - -The prepared session is an optimization, not shared conversational memory. A request consumes the -compatible prewarmed session once; field or context resets discard state that could contaminate a new -suggestion. - -Framework or language unavailability is classified distinctly from cancellation and ordinary -generation failure so the router can apply its narrow fallback policy. - -## In-Process llama - -LlamaSuggestionEngine converts the request into the base prompt, invokes the runtime manager, maps -runtime errors into suggestion errors, and passes token-stream partials through normalization before -they reach the coordinator. - -LlamaRuntimeManager is a MainActor ObservableObject. It publishes model discovery, load, warmup, -generation, and failure state for UI consumers. It does not perform tokenization or decoding on the -MainActor. Heavy calls are dispatched away from UI isolation and delegated to LlamaRuntimeCore. - -LlamaRuntimeCore is a lock-protected nonisolated class around mutable native state, not a Swift -actor. Its boundaries are explicit: - -- A lifecycle condition coordinates load, active operations, and bounded shutdown. -- An autocomplete lock serializes prompt-cache and decode state. -- Native pointers and CotabbyInference calls remain private to the core. -- Cancellation is checked through an abort callback while native decoding is running. -- Shutdown prevents new operations, waits for in-flight work, then releases model and context state. - -CotabbyInference is an external SwiftPM wrapper around llama.cpp. It is consumed as a package rather -than vendored in this repository. Pointer ownership, Metal buffers, token buffers, and llama context -correctness stay below the manager boundary. - -## Prompt Cache and Sampling - -The llama core tokenizes the rendered prompt, compares it with the previously evaluated token -sequence, reuses the longest safe common prefix in the KV cache, prefills the remaining prompt, and -then samples new tokens. Token callbacks make partial output available while decode continues. - -Cache reuse is a latency optimization constrained by request continuity. Resetting backend context, -loading another model, or changing to an incompatible prompt invalidates reuse. The cache must never -turn one field's text into hidden context for another field. - -Sampling and stop decisions live inside the native runtime boundary. Output safety still does not: -every backend passes text through the same normalizer before Cotabby presents or inserts it. - -## Runtime Memory Lifecycle - -The local runtime is started only when the selected engine is Open Source. Switching to Apple or an -endpoint stops it so mapped GGUF weights and Metal buffers are released. A model selection change, -download completion, or model-directory refresh can trigger a reload followed by warmup when the -active engine needs it. - -This policy prevents an unused local model from consuming memory while another backend is active. -The manager publishes lifecycle state; AppDelegate decides when engine selection permits the process -to own native resources. - -## OpenAI-Compatible Endpoints - -The endpoint backend supports completion-style and chat-style OpenAI-compatible APIs, including SSE -stream parsing. The configured endpoint may be: - -- Loopback, such as the default Ollama address at http://127.0.0.1:11434/v1 -- A server on the local network -- A public HTTPS service - -Public insecure HTTP is rejected. Configuration models classify endpoint privacy and surface a -warning when text may leave the device. API credentials are stored in Keychain through -OpenAICompatibleCredentialStore rather than in UserDefaults. - -Ollama model discovery and preload are endpoint-specific conveniences. Preload work is coalesced so -repeated settings or warmup events do not launch redundant load requests. Endpoint connection state -is invalidated when URL, model identity, request mode, or credential changes. - -The endpoint path means Cotabby is local-first, not unconditionally offline. Privacy documentation -and UI must distinguish on-device engines from a user-configured remote server. - -## Streaming Contract - -Engines emit cumulative partial text. Each partial is normalized before the coordinator sees it, and -the coordinator accepts only monotonic extensions suitable for replacing the currently visible ghost -tail. UI rendering is coalesced independently of backend token cadence. - -Cancellation is expected when the user types or switches focus. It is not logged or presented as a -runtime failure. A final result is authoritative even after partials were shown; final normalization -or seam checks can replace or suppress the provisional text. - -## Backend-Independent Normalization - -SuggestionTextNormalizer is intentionally outside all engines. It handles: - -- Control and special-token residue -- Reasoning blocks and model scaffolding -- Prompt or prefix echo -- Leading-whitespace reconciliation -- Single-line versus bounded multiline policy -- Text already present after the caret -- Repeated trailing-prefix material -- Empty, unsafe, or non-insertable results - -A new backend must satisfy this same output contract. Backend-specific code should not create a -parallel set of cleanup semantics unless the shared normalizer first receives a genuine domain rule. - -## Diagnostics - -Every request carries a request ID through coordinator, router, engine, runtime, and LLM-I/O logs. -The debug prompt payload comes from the same request factory used for generation, so developer logs -show the actual selected-backend context shape without engines reaching into UI state. It is not -part of the Settings UI or normal user-facing state. - -OSLog metadata is available in normal operation. Full prompts and completions are written to the -debug JSONL sink only when the application is launched with -cotabby-debug. That switch is important -because prompt content can include user text and optional screen or clipboard context. - -## Invariants - -- Routing selects an engine; it does not let engines read live editor state. -- The request is immutable once asynchronous generation starts. -- Base GGUF prompts are continuation-shaped; Apple prompts use its instructions channel. -- The caret prefix is preserved ahead of optional context under budget pressure. -- Heavy llama work never runs on the MainActor. -- Llama native state is serialized under the core's explicit locks and lifecycle condition. -- Context reset prevents cache data from crossing interaction boundaries. -- Switching away from Open Source releases local runtime resources. -- Endpoint secrets live in Keychain and insecure public HTTP is rejected. -- All backends share normalization and streaming semantics. -- Cancellation is a normal lifecycle outcome, not automatically a generation error. - -## Failure-Oriented Reading - -- Wrong engine runs: router selection and settings/profile snapshot. -- Apple unexpectedly falls back: language-support classification in the foundation engine. -- First suggestion is slow: selected-backend prewarm and llama prompt prefill/KV reuse. -- Memory stays high on endpoint mode: AppDelegate runtime stop and manager shutdown. -- Output includes instructions or prompt text: renderer choice and shared normalizer. -- Old field text influences a new field: reset broadcast and llama cache continuity. -- Endpoint never streams: request mode, SSE parsing, and cumulative-partial contract. -- Local endpoint warning looks remote or vice versa: endpoint privacy classification. -- UI freezes during generation: manager-to-core dispatch and synchronous work on MainActor. -- Shutdown crashes in native code: core lifecycle condition and process termination order. - -## Update This Guide When - -Update this document when a backend is added, router fallback policy changes, request fields or -budgets change, prompt shape changes, runtime serialization changes, endpoint privacy rules change, -or streaming and normalization acquire a new cross-backend contract. diff --git a/.internal/architecture/input-and-insertion.md b/.internal/architecture/input-and-insertion.md deleted file mode 100644 index bdca722a..00000000 --- a/.internal/architecture/input-and-insertion.md +++ /dev/null @@ -1,199 +0,0 @@ -# Input and Insertion - -## Purpose - -This guide explains how Cotabby observes global input, temporarily intercepts only the keys it owns, -and commits accepted text into another application. This is a process-wide boundary: mistakes can -drop user keystrokes, feed Cotabby's synthetic events back into its own state machine, or disturb the -user's clipboard. The design therefore separates observation, consumption, suppression, policy, and -insertion. - -## Main Files - -Read these files in order: - -1. [InputMonitor.swift](../../Cotabby/Services/Input/InputMonitor.swift) -2. [InputSuppressionController.swift](../../Cotabby/Services/Input/InputSuppressionController.swift) -3. [SuggestionInserter.swift](../../Cotabby/Services/Suggestion/SuggestionInserter.swift) -4. [InsertionStrategySelector.swift](../../Cotabby/Support/Suggestion/InsertionStrategySelector.swift) -5. [InsertionSafetyGate.swift](../../Cotabby/Support/Suggestion/InsertionSafetyGate.swift) - -InputMonitor owns event-tap lifetime and converts Core Graphics keyboard events into semantic -CapturedInputEvent values. SuggestionInserter owns the write boundary. The two pure Support policies -decide whether insertion is safe and which mechanism is appropriate without touching global state. - -## Three Event-Tap Responsibilities - -InputMonitor deliberately does not use one permanently consuming event tap. - -~~~text -steady listen-only observer - -> classify ordinary typing, deletion, navigation, and pointer activity - -conditional consuming tap - -> active only while a suggestion or inline-command capture needs interception - -> consume only an event the owning coordinator successfully handles - -dedicated global-toggle tap - -> active only when the shortcut is configured - -> decide the process-wide toggle independently of suggestion visibility -~~~ - -The observer tap is installed at the head of the event chain with listen-only behavior. It can see -events without delaying the focused application or swallowing a key. This is the steady-state path. - -The conditional tap is a default tap because returning nil is the only way to keep an accepted key -from also reaching the host. It is installed only when a visible suggestion claims acceptance keys -or when the emoji picker has an active capture. All unrelated keys pass through unchanged. - -The global-toggle shortcut has a separate consuming tap. Its ownership does not depend on an active -suggestion, and separating it prevents suggestion tap lifetime from changing whether the global -shortcut works. - -## Fail-Open Consumption - -Matching a configured key is not sufficient reason to swallow it. The consuming tap first asks the -current owner to handle the event. It returns nil only when acceptance or capture succeeds. A stale -tap, vanished overlay, changed focus, revoked permission, or rejected session causes the original key -to pass through to the focused application. - -Word/phrase acceptance and full-tail acceptance have independent configurable key and modifier -bindings. Full acceptance takes priority if the bindings overlap. The event is classified using the -current settings at event time so changing a shortcut does not require rebuilding all tap behavior. - -After a final acceptance hides the overlay, the accept tap lingers for a very short deferred teardown. -The callback that consumed the physical key may still be posting synthetic insertion events. Removing -its mach port immediately can prevent the last accepted chunk from reaching the host. The delayed -teardown rechecks whether a suggestion or capture has re-armed the tap before removing it. - -Exhausting a tail also creates a bounded regeneration window in which a rapid follow-up Tab should -not leak into the host and move focus. PostExhaustionAcceptanceState owns the pure arm, queue, -consume, and timeout-generation rules. Repeated presses collapse to one queued accept; the -coordinator owns the timer and interception side effects and returns the key to the host when the -window expires or the session tears down. - -## Inline-Command Capture - -Emoji capture shares the conditional tap but not suggestion acceptance semantics. While a colon -query is active, the emoji controller decides whether navigation, dismissal, or commit keys are -handled. The listen-only observer still routes the event to the coordinator, and the consuming tap -swallows it only after the emoji path reports success. - -This ordering lets Tab remain a configurable suggestion accept key while also committing an emoji -selection during capture. InputMonitor knows which subsystem currently owns interception; it does not -implement emoji query or suggestion-session rules itself. - -## Suppressing Cotabby's Own Events - -[InputSuppressionController.swift](../../Cotabby/Services/Input/InputSuppressionController.swift) -prevents inserted text from being mistaken for fresh user typing. SuggestionInserter marks every -synthetic Core Graphics event with a Cotabby-specific source value and registers an expected event -burst before posting it. - -The marker is the strongest identity check. The bounded suppression countdown is a compatibility -fallback for event transformations that do not preserve every property. It accumulates across rapid -acceptances because global event delivery is asynchronous; overwriting the count could expose the -tail of an earlier insertion as apparent user input. - -Suppression is intentionally narrow and expires. It must not hide real typing after an insertion or -become a general pause in focus reconciliation. - -## Pre-Insertion Safety - -The coordinator validates the active session, overlay, focus, and current text before asking for a -write. InsertionSafetyGate provides the pure final checks for the proposed chunk. Invalid or unsafe -text is rejected before an event is posted. - -This boundary does not assume that generation success authorizes insertion. The user may have moved -the caret, selected text, switched applications, or edited the field after the suggestion became -visible. - -## Default Unicode Keystroke Path - -For the common short, single-line continuation, SuggestionInserter posts a Unicode key-down/key-up -pair through Core Graphics. This path is app-agnostic and does not touch the clipboard. It also avoids -relying on an application's support for direct Accessibility value mutation. - -The events use a placeholder virtual key because the committed payload is carried as Unicode text. -That detail is why the synthetic marker matters: a user could bind acceptance to the same key code, -and Cotabby must never consume its own insertion as another acceptance. - -## IME-Safe and Optional Paste Paths - -Synthetic Unicode events can be intercepted as composing input while an input method editor is -active. In that state, SuggestionInserter commits through a clipboard paste so accepted Japanese, -Chinese, or other composed text lands as final text rather than being fed back into composition. - -There is also a default-off paste strategy for long or multiline suggestions. When enabled, -InsertionStrategySelector chooses paste for multiline text or chunks at least 80 characters long. -Short ordinary completions remain on the clipboard-free Unicode path. - -The paste path: - -1. Snapshots every representation of every item on the general pasteboard. -2. Replaces the pasteboard temporarily with the accepted plain text. -3. Tries the target application's Accessibility Edit > Paste menu action first. -4. Falls back to a synthetic Command-V when the menu item cannot be pressed. -5. Restores the saved clipboard after the host has had time to service the paste. - -Pressing the actual Paste menu item is important for applications such as Chrome, where a synthetic -Command-V posted from the global tap callback can fail while a physical accept key is still down. -The menu item is cached per process only after its AXPress result is validated. - -Clipboard restoration is change-count aware. If the user or another application changes the -clipboard before restoration, Cotabby leaves the newer contents alone. Overlapping paste insertions -reuse the one saved user clipboard instead of accidentally snapshotting Cotabby's previous completion -and restoring that to the user. - -## Replacement Writes - -A normal suggestion inserts a continuation at the caret. A spelling-correction or inline-command -session can instead replace a known literal run. The inserter posts the required deletion burst and -then the replacement text under the same suppression boundary. - -Replacement length is derived from the validated session, not from a new untrusted scan at write -time. The coordinator still reconciles the host's later AX publication because posting events proves -only that Cotabby requested the mutation, not that every application applied it exactly as expected. - -## Permissions and Recovery - -Input Monitoring permission governs global event observation. Accessibility is also required for -focus validation and for the preferred paste-menu action. If taps are disabled by timeout or user -input, InputMonitor re-enables them where safe. If permission is revoked, the consuming tap is torn -down and later recreated only after the permission state allows it. - -No recovery path may leave a dead consuming tap installed. A tap that cannot confidently handle an -event must pass it through. - -## Invariants - -- Steady-state observation is listen-only. -- A key is consumed only after the owning feature reports success. -- Post-exhaustion Tab ownership is bounded, generation-keyed, and can queue at most one accept. -- The consuming suggestion/capture tap exists only while interception is needed. -- Global toggle ownership is independent of suggestion visibility. -- Every synthetic event is tagged and covered by bounded self-event suppression. -- Short ordinary completions use the clipboard-free Unicode path. -- IME-active insertion uses a commit mechanism that bypasses composition. -- Paste restores all saved pasteboard representations unless newer clipboard activity wins. -- Insertion never trusts a generation result without revalidating the live session. -- Host publication after insertion is reconciled rather than assumed. - -## Failure-Oriented Reading - -- Acceptance key is swallowed with no visible suggestion: fail-open authorization in InputMonitor. -- Accepted text triggers another prediction as typing: synthetic marker and suppression accounting. -- Final accepted chunk disappears: deferred accept-tap teardown and event posting. -- Rapid Tab moves host focus after the visible tail ends: post-exhaustion acceptance window and - backstop generation. -- Tab commits the suggestion instead of emoji: capture ownership and accept-tap resolution order. -- IME acceptance regenerates or enters composition: active-input-source detection and paste path. -- Clipboard contains a completion afterward: snapshot, overlap, change-count, and restore logic. -- Chrome ignores paste: AX Paste menu lookup before Command-V fallback. -- Correction removes the wrong characters: replacement session validation and deletion count. - -## Update This Guide When - -Update this document when an event tap gains a new ownership reason, acceptance bindings change -semantics, suppression identity changes, a new insertion strategy is added, clipboard restoration -policy changes, or another inline feature begins consuming global input. diff --git a/.internal/architecture/lifecycle-and-composition.md b/.internal/architecture/lifecycle-and-composition.md deleted file mode 100644 index 4183485e..00000000 --- a/.internal/architecture/lifecycle-and-composition.md +++ /dev/null @@ -1,195 +0,0 @@ -# Lifecycle and Composition - -## Purpose - -This guide explains who creates Cotabby's long-lived objects, who starts and stops them, and why -views do not construct services. It is the first deep dive to read after the root architecture map. - -Cotabby integrates with process-wide macOS facilities: Accessibility, global event taps, screen -capture, Sparkle, and an in-process native model runtime. Accidentally constructing two copies of -one of those services can produce duplicate observers, competing event taps, runtime reload races, -or two pieces of UI that disagree about current settings. The composition root exists to prevent -that class of bug. - -## Primary Owners - -The lifecycle is divided among three files: - -1. [CotabbyApp.swift](../../Cotabby/App/Core/CotabbyApp.swift) declares the SwiftUI scene graph and - bridges the AppKit delegate into the SwiftUI application. -2. [AppDelegate.swift](../../Cotabby/App/Core/AppDelegate.swift) owns process lifecycle callbacks, - starts and stops services, and wires reactions that are specifically tied to application - lifecycle. -3. [CotabbyAppEnvironment.swift](../../Cotabby/App/Core/CotabbyAppEnvironment.swift) constructs the - dependency graph once and retains app-scoped subscriptions between the objects it created. - -The split is intentional. Construction answers "which concrete objects satisfy the app's -dependencies?" Lifecycle answers "when may those objects begin side effects?" SwiftUI answers -"which state should be presented?" Keeping those questions separate makes ownership visible. - -## Process Startup - -CotabbyApp is the process entry point. It creates AppDelegate through NSApplicationDelegateAdaptor -and declares the MenuBarExtra scene. SwiftUI owns insertion and removal of the status item, while -SuggestionSettingsModel remains the durable source of truth for whether the icon is visible. - -AppDelegate initialization performs synchronous graph construction: - -~~~text -CotabbyApp - -> AppDelegate.init - -> CotabbyLogger.bootstrap - -> CotabbyAppEnvironment.init - -> expose shared environment objects to SwiftUI - -> install cross-subsystem closures and subscriptions -~~~ - -Constructing the graph does not start global event processing. AppDelegate waits for -applicationDidFinishLaunching before it: - -1. Applies the one-time Open at Login default. -2. Starts the llama runtime only when the selected engine requires it. -3. Starts focus polling. -4. Starts global input monitoring. -5. Starts the update manager. -6. Starts the suggestion coordinator. -7. Starts the inline-command coordinator. -8. Presents onboarding or permission reminders when needed. -9. Restores a visible Settings surface when the menu bar icon is hidden and launch policy requires - recovery. - -The XCTest host is a special case. App-hosted unit tests launch the real application binary before -loading the test bundle. AppDelegate detects that environment and skips production service startup -so tests do not install global taps, begin polling, launch Sparkle, or load a model as a side effect. - -## The Environment Graph - -CotabbyAppEnvironment is MainActor-isolated because most graph members publish UI state, call -AppKit, or access Accessibility. It constructs shared services in dependency order: - -~~~text -settings and permission state - -> input/focus services - -> insertion and presentation services - -> context services - -> generation engines and router - -> suggestion coordinator - -> emoji and macro controllers - -> settings/onboarding coordinators -~~~ - -Important shared instances include: - -- PermissionManager and PermissionGuidanceController -- SuggestionSettingsModel -- LlamaRuntimeManager wrapped by RuntimeBootstrapModel -- ModelDownloadManager -- FocusTrackingModel and InputMonitor -- InputSuppressionController and SuggestionInserter -- OverlayController and ActivationIndicatorController -- ClipboardContextProvider and VisualContextCoordinator -- SuggestionEngineRouter with all generation backends -- SuggestionCoordinator, SuggestionInteractionState, and SuggestionWorkController -- EmojiPickerController, MacroController, and InlineCommandCoordinator -- SettingsCoordinator and WelcomeCoordinator - -The environment passes narrow protocol-shaped collaborators into SuggestionCoordinator. This keeps -the coordinator testable without turning every concrete service into a global singleton. - -## Where Subscriptions Live - -Both the environment and AppDelegate retain Combine subscriptions. That is not accidental -duplication; the dividing line is ownership. - -CotabbyAppEnvironment retains relationships among objects it constructed: - -- Settings changes update the focus poll interval. -- Binding or clearing the global-toggle shortcut installs or removes its dedicated event tap. -- Power-source and profile changes select the appropriate engine and model. -- Selecting the endpoint engine triggers model discovery. -- Endpoint identity or credential changes invalidate connection state. - -AppDelegate retains process-lifecycle reactions: - -- Permission changes refresh input monitoring. -- Engine changes start or release the in-process llama runtime. -- Focus changes update activation and debug overlays. -- Settings changes hide the activation indicator when Cotabby becomes disabled or paused. -- Model-directory changes refresh available runtime models. -- Visual-context state is mirrored into the debug overlay. - -CotabbyAppEnvironment itself must therefore be retained for the whole process. If it were a -temporary local variable in AppDelegate.init, its cancellables would deallocate and settings-driven -behavior would silently stop. - -## Runtime Selection and Memory Ownership - -The in-process llama runtime is warmed only when the selected engine is Open Source. Selecting Apple -Intelligence or an OpenAI-compatible endpoint stops the local runtime so mapped model weights and -Metal buffers are released. Model downloads or manual file changes trigger a rescan, followed by a -warmup only if the current engine needs llama. - -Power-based profiles are applied in the environment. A profile can select Apple Intelligence, -an installed GGUF model, or an endpoint model. Applying a profile is idempotent so repeated -publisher emissions do not cause unnecessary model reloads. - -## Shutdown - -applicationWillTerminate reverses process-wide side effects: - -1. Hide activation and debug overlays. -2. Stop suggestion and inline-command coordination. -3. Stop input monitoring. -4. Stop focus polling. -5. Synchronously release the native llama runtime with a bounded wait. - -The synchronous native shutdown is a correctness requirement. Exiting while llama/Metal resources -remain alive can collide with C++ static destruction. At the same time, shutdown cannot defer -application termination indefinitely because macOS permission flows rely on a prompt quit and -relaunch to observe a new TCC grant. - -## Settings and Presentation Ownership - -SuggestionSettingsStore persists durable non-secret values in UserDefaults. -SuggestionSettingsModel keeps individually `@Published` properties for SwiftUI and source -compatibility. Its `domainSettings` projection builds a `SuggestionSettingsData` value grouped into -general, engine, completion, context, correction, presentation, inline-feature, and shortcut -domains. The projection does not duplicate storage or change publication timing. - -SuggestionSettingsSnapshot is derived from those domains and remains the immutable behavior input -for the suggestion pipeline. SuggestionSettingsStore deliberately maps the domain values back to -the existing flat UserDefaults keys so the refactor does not migrate or invalidate persisted user -preferences. OpenAI-compatible credentials are kept in Keychain rather than UserDefaults. - -Views observe shared models or receive callbacks from coordinators. They do not instantiate focus, -input, runtime, permission, download, or update services. AppKit window controllers are likewise -constructed at app scope when their lifetime must outlive a SwiftUI view redraw. - -## Invariants - -- There is one production instance of each process-wide observer, tap, coordinator, and runtime. -- Construction does not imply startup; AppDelegate controls when side effects begin. -- SwiftUI view reconstruction must not reconstruct services. -- The environment remains retained as long as its subscriptions are needed. -- Heavy model, OCR, download, and file-copy work must not block the MainActor. -- Domain settings are projections over one durable source, not a second mutable settings graph. -- Switching away from llama releases native runtime resources. -- Production services do not start inside the XCTest host. -- Shutdown stops new work before releasing native state. - -## Failure-Oriented Reading - -- Duplicate taps or observers: start in CotabbyAppEnvironment and AppDelegate initialization. -- A setting changes but behavior does not: inspect the environment's retained cancellables. -- Runtime remains loaded on another engine: inspect AppDelegate engine switching. -- Hidden menu bar icon makes the app unreachable: inspect MenuBarRecoveryPolicy and reopen handling. -- Permission relaunch loops or termination crashes: inspect AppDelegate termination and permission - guidance. -- A view appears to own a service: trace construction back to CotabbyAppEnvironment before changing - view lifetime. - -## Update This Guide When - -Update this document when a new app-lifetime service is added, startup or shutdown order changes, -ownership moves between AppDelegate and CotabbyAppEnvironment, a new engine affects runtime memory -policy, or a SwiftUI scene begins owning behavior instead of presentation. diff --git a/.internal/architecture/presentation-and-sibling-features.md b/.internal/architecture/presentation-and-sibling-features.md deleted file mode 100644 index 3776240b..00000000 --- a/.internal/architecture/presentation-and-sibling-features.md +++ /dev/null @@ -1,241 +0,0 @@ -# Presentation and Sibling Features - -## Purpose - -This guide covers Cotabby-owned UI that appears around another application's editor and the features -that share global input infrastructure with suggestions. The presentation layer is a deliberate mix -of SwiftUI and AppKit: SwiftUI describes content, while AppKit owns non-activating panels, window -levels, focus behavior, and geometry that ordinary SwiftUI scenes cannot express reliably. - -## Presentation Ownership - -The main suggestion presentation path is: - -~~~text -SuggestionCoordinator - -> SuggestionOverlayPresenter - -> OverlayController - -> non-activating NSPanel - -> SwiftUI ghost or mirror content -~~~ - -[SuggestionOverlayPresenter.swift](../../Cotabby/Services/Suggestion/SuggestionOverlayPresenter.swift) -adapts coordinator intent into show, move, update, or hide actions and returns diagnostic reasons. -It owns the small state-diff rules, not AppKit window construction. - -[OverlayController.swift](../../Cotabby/Services/Presentation/OverlayController.swift) owns the reusable -borderless panel, SwiftUI hosting views, render-mode selection, layout, font and color resolution, -fade behavior, and visible OverlayState. It is the only owner allowed to treat the suggestion as an -AppKit window. - -The panel is non-activating, ignores mouse events, joins all Spaces, can appear over full-screen apps, -and does not participate in the window cycle. Showing a suggestion must never take keyboard focus -from the editor Cotabby is assisting. - -## Inline and Mirror Modes - -[CompletionRenderModePolicy.swift](../../Cotabby/Support/Presentation/CompletionRenderModePolicy.swift) chooses -how to render from user preference and geometry quality. - -- Inline mode paints ghost text immediately beside a trusted caret. Exact and derived geometry are - precise enough for this path under automatic policy. -- Mirror mode presents a Cotabby-owned card near the text line when geometry is estimated, - layout-estimated, intentionally forced by user preference, or the caret is mid-line where free - ghost glyphs could overlap existing trailing text. - -The mirror card is not pretending to be host text. It trades visual continuity for correctness when -the exact glyph insertion point is unknown. MirrorOverlayLayout uses the reason for promotion to -choose an anchor and keep the card inside the visible screen. - -Per-application render-mode overrides are represented as future policy capacity but are not wired in -the current product. Documentation should not claim they ship until the focused bundle identifier is -actually passed to the policy. - -## Field-Matched Rendering - -Focus resolution can carry a ResolvedFieldStyle with font family, weight, size, and foreground color. -Inline rendering uses that style when available and derives size primarily from caret height. It -falls back to a system font when the host exposes no usable style. - -Presentation also supports: - -- User-selected ghost color and opacity -- A bounded size multiplier -- Configurable acceptance-key hint -- Optional fade-in duration, respecting Reduce Motion -- Green correction styling for replace-the-word suggestions -- Right-to-left placement -- Bounded multiline layout and wrapping inside the field or screen - -These are presentation settings. They do not enter SuggestionRequest or change generated text. - -## Stable Partial and Acceptance Updates - -Streaming can update many times per second. The coordinator coalesces partials and the presenter -skips identical text/geometry. OverlayController reuses separate typed NSHostingView instances for -inline and mirror roots so token extensions do not allocate a new window or hosting hierarchy. - -For a single-line left-to-right inline suggestion, word acceptance and exact type-through can advance -the remaining tail by the measured width of the committed prefix. That keeps the ghost visually -stationary relative to the user's typing instead of waiting for a noisy AX caret refresh. - -[SuggestionOverlayStabilityGate.swift](../../Cotabby/Support/Presentation/SuggestionOverlayStabilityGate.swift) -decides whether later reconciliation should hold the existing anchor or re-anchor to fresh geometry. -It tolerates the known pre-insertion AX frame briefly and corrects meaningful drift without allowing -small polling noise to make the tail jitter. - -Mirror and multiline cases fall back to fresh layout when a simple horizontal advance is not valid. - -## Activation and Debug Overlays - -[ActivationIndicatorController.swift](../../Cotabby/Services/Presentation/ActivationIndicatorController.swift) -owns the optional caret or field-edge availability indicator. It follows supported focus and is -hidden when Cotabby is disabled, paused, missing required permissions, or not eligible. - -[FocusDebugOverlayController.swift](../../Cotabby/Services/Presentation/FocusDebugOverlayController.swift) -shows lightweight polling sequence/capability, caret and field geometry, build identity, and -visual-context status for development. The old detailed FocusInspectionSnapshot and published -suggestion-diagnostics surface are not part of this path. It is gated by -cotabby-debug and must not -become normal user-facing settings state. - -These controllers use their own non-activating panels because their visibility and layout lifetimes -are independent from suggestion text. - -## Inline Command Ownership - -[InlineCommandCoordinator.swift](../../Cotabby/App/Coordinators/InlineCommandCoordinator.swift) fans -captured input to the emoji and macro controllers. InputMonitor exposes one capture-decision slot and -one capture-interception flag; the coordinator prevents the two features from competing for those -shared resources. - -Emoji and macro sigils are disjoint, so at most one capture should be active. The coordinator routes -the consuming-tap decision to the current owner and reports whether either feature was involved so -SuggestionCoordinator can stand down for that event. - -Both features: - -- Check live settings before triggering. -- Require a supported, non-secure focus context. -- Pin capture to a focus-change sequence. -- Cancel on incompatible navigation, focus change, timeout, or dismissal. -- Replace the literal typed run through SuggestionInserter. -- Use pure Support state machines for keystroke-to-action rules. - -## Emoji Picker - -[EmojiPickerController.swift](../../Cotabby/App/Coordinators/EmojiPickerController.swift) owns one -colon-query capture. [EmojiTriggerStateMachine.swift](../../Cotabby/Support/Emoji/EmojiTriggerStateMachine.swift) -decides when a colon may open a query and how typing, deletion, navigation, closing colon, acceptance, -and dismissal change the capture. - -[EmojiCatalog.swift](../../Cotabby/Support/Emoji/EmojiCatalog.swift) lazily loads the bundled resource. -[EmojiMatcher.swift](../../Cotabby/Support/Emoji/EmojiMatcher.swift) ranks query matches using names, -keywords, synonyms, popularity, recency, and usage without owning UI. - -[EmojiPickerPanelController.swift](../../Cotabby/Services/Presentation/EmojiPickerPanelController.swift) owns the -non-activating picker panel and SwiftUI content. Arrow keys move selection, the configured word-accept -key commits a match, and a supported closing-colon form can commit the best match. The controller then -replaces the literal colon query and records recency/frequency in EmojiUsageStore. - -Catalog and matcher initialization are lazy so users who never type an emoji query do not pay the -resource and index cost at launch. - -## Macros - -[MacroController.swift](../../Cotabby/App/Coordinators/MacroController.swift) owns slash-query capture -and one-row preview presentation. [MacroTriggerStateMachine.swift](../../Cotabby/Support/Macros/MacroTriggerStateMachine.swift) -defines capture semantics, including guards that prevent ordinary URL fragments, fractions, and -mid-word slashes from triggering. - -[MacroEngine.swift](../../Cotabby/Support/Macros/MacroEngine.swift) tries deterministic evaluator -families in priority order: - -- Date and time expressions -- Random values -- Unit conversion -- Currency conversion -- Arithmetic - -Clock, calendar, locale, and random source are injectable, so the engine remains deterministic under -test. A successful result carries preview text and replacement text; accepting it replaces the typed -slash query rather than invoking any language model. - -## Settings - -[SettingsCoordinator.swift](../../Cotabby/App/Coordinators/SettingsCoordinator.swift) owns the single -AppKit settings window and hosts [SettingsContainerView.swift](../../Cotabby/UI/Settings/SettingsContainerView.swift). -The window survives SwiftUI view reconstruction, remembers placement through AppKit, and routes -permission actions to PermissionGuidanceController. - -Settings panes under Cotabby/UI/Settings/Panes remain presentation-focused. They bind to shared -models and call narrow mutation methods. Search indexing, attention callouts, hardware recommendations, -and validation rules belong in Support or Models so the view hierarchy does not become the source of -product behavior. - -SuggestionSettingsModel retains individually published properties for existing SwiftUI bindings. -Its domainSettings projection groups the same values by the subsystem that owns each decision, and -SuggestionSettingsSnapshot supplies the smaller immutable behavior surface used by the pipeline. -SuggestionSettingsStore remains the sole UserDefaults owner and keeps persisted keys flat for -compatibility. - -ContextLivePreviewField is the one Cotabby-owned editable field permitted through focus capability -for live preview. The exception is identified explicitly; general Cotabby windows remain blocked from -autocomplete to prevent the app from assisting itself. - -## Onboarding - -[WelcomeCoordinator.swift](../../Cotabby/App/Coordinators/WelcomeCoordinator.swift) owns the onboarding -and required-permission reminder windows. It restores incomplete progress, presents the correct step, -resizes the AppKit window to SwiftUI content, and keeps permission guidance attached to the active -surface. - -Onboarding templates recommend a settings configuration; they do not create a separate runtime or -coordinator graph. Applying a template mutates the shared settings model so the same lifecycle -subscriptions and validation rules run as they would for manual changes. - -Input Monitoring can require a quit/relaunch before the new TCC state is effective. WelcomeCoordinator -persists progress and returns the user to the permission step rather than treating that process exit -as abandoned onboarding. - -## Menu Bar and Reachability - -[CotabbyApp.swift](../../Cotabby/App/Core/CotabbyApp.swift) declares the SwiftUI MenuBarExtra. Status -item visibility is persisted in shared settings. Because hiding the status item can make an agent app -hard to reach, AppDelegate and MenuBarRecoveryPolicy restore a settings surface under defined launch -or reopen conditions. - -This is a lifecycle rule, not merely a view preference: any new way to hide the menu icon must retain -a reliable path back into Settings. - -## Invariants - -- Cotabby-owned panels never steal focus from the assisted editor. -- SuggestionOverlayPresenter decides presentation actions; OverlayController owns AppKit mechanics. -- Automatic mode uses inline only when geometry and editing position make it safe. -- Streaming updates reuse presentation objects and avoid redundant renders. -- Partial acceptance updates the remaining tail synchronously. -- Debug presentation remains behind the explicit debug launch gate. -- Emoji and macro capture never compete for the one consuming-tap slot. -- Inline commands require a supported non-secure field and stay pinned to one focus sequence. -- Pure query and evaluation rules remain outside AppKit controllers. -- Settings and onboarding observe the app-scoped graph rather than constructing services. -- Hiding the menu bar item never permanently removes access to settings. - -## Failure-Oriented Reading - -- Ghost steals application focus: OverlayPanel configuration and show calls. -- Ghost overlaps host text mid-line: completion render-mode policy and geometry flags. -- Card appears despite exact caret: user mirror preference and quality passed into policy. -- Tail jitters after acceptance: advanceInline and SuggestionOverlayStabilityGate. -- Wrong font or color: resolved field style capture and overlay fallback. -- Emoji and suggestion both accept Tab: InlineCommandCoordinator capture ownership. -- Colon in a URL opens the picker: EmojiTriggerStateMachine boundary rules. -- Slash in a fraction opens macros: MacroTriggerStateMachine boundary rules. -- Settings opens duplicate windows: SettingsCoordinator lifetime. -- Onboarding restarts from the wrong step: WelcomeCoordinator persisted progress. -- Hidden icon leaves no entry point: MenuBarRecoveryPolicy and application reopen handling. - -## Update This Guide When - -Update this document when a render mode, panel, inline feature, settings surface, onboarding step, or -menu-bar reachability rule is added or changes ownership. diff --git a/.internal/architecture/suggestion-pipeline.md b/.internal/architecture/suggestion-pipeline.md deleted file mode 100644 index e233538c..00000000 --- a/.internal/architecture/suggestion-pipeline.md +++ /dev/null @@ -1,265 +0,0 @@ -# Suggestion Pipeline - -## Purpose - -This guide follows one autocomplete attempt from an input event to visible or inserted text. The -pipeline is a state machine operating over eventually consistent Accessibility data. Reliability -comes from explicit work identity, cancellation, focus signatures, session reconciliation, and -safe failure behavior rather than from assuming that events arrive in a convenient order. - -## Coordinator Reading Order - -Read the coordinator in this order: - -1. [SuggestionCoordinator.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator.swift) -2. [SuggestionCoordinator+Lifecycle.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator+Lifecycle.swift) -3. [SuggestionCoordinator+Input.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator+Input.swift) -4. [SuggestionCoordinator+Prediction.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator+Prediction.swift) -5. [SuggestionCoordinator+Acceptance.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator+Acceptance.swift) - -The base file declares dependencies and mutable orchestration state. The extensions group lifecycle, -input, prediction, and acceptance behavior without turning those concerns into separate owners. - -## End-to-End Flow - -~~~text -focus snapshot or input event - -> availability and session reconciliation - -> debounce and work identity - -> refresh AX state if stale - -> native correction decision - -> bounded request construction - -> selected engine generation - -> streamed partial presentation - -> authoritative final normalization and guards - -> active suggestion session - -> overlay presentation - -> type-through, dismissal, partial acceptance, or full acceptance - -> host publish reconciliation and next prediction -~~~ - -## Starting and Stopping - -start installs callbacks on the focus provider, input monitor, permissions, settings, overlay, and -visual-context coordinator. It also snapshots current settings and begins reacting to the current -focus. stop removes or neutralizes those callbacks, cancels all prediction work, clears active -sessions, hides presentation, and resets backend-local generation context. - -The coordinator is MainActor-isolated. That makes transitions involving coordinator state, AppKit -presentation, and Accessibility snapshots sequential from the coordinator's perspective. Only -state that UI actually observes remains `@Published`; internal debug/session values are plain -MainActor properties. Heavy backend work is moved off the actor by the backend implementations. - -## Focus Changes and Prewarm - -A supported focused field creates or refreshes a field-scoped interaction context. Focus changes: - -- Cancel obsolete generation work. -- Reset active suggestion and backend continuation state when continuity is broken. -- Start visual-context capture for the new field when enabled and permitted. -- Build a request-shaped warmup payload. -- Prewarm only the selected backend. - -Prewarm is opportunistic. Failure to warm never becomes a user-facing prediction failure; it only -means the first real request pays the cold setup cost. - -## Input Handling - -Inline commands receive first look at captured input. If the emoji or macro feature owns the current -keystroke, normal suggestion behavior stands down and any conflicting ghost text is hidden. - -For ordinary input, the coordinator distinguishes: - -- Direct text mutation -- Deletion -- Navigation or selection movement -- Escape/dismissal -- Word acceptance -- Full acceptance -- Synthetic events Cotabby generated itself - -Direct typing can advance a visible session without regenerating when the typed characters exactly -match the next suggestion tail. Divergent typing invalidates the session and schedules new work. -Navigation, selection, focus changes, or incompatible trailing-text changes clear stale sessions. - -## Work Identity and Cancellation - -[SuggestionWorkController.swift](../../Cotabby/Services/Suggestion/SuggestionWorkController.swift) -owns the debounce task, generation task, and a monotonically increasing work ID. - -Replacing work: - -1. Cancels the previous debounce and generation tasks. -2. Advances the work ID. -3. Starts a new debounce operation. -4. Allows generation to apply results only while that ID remains current. - -Cancellation alone is insufficient because native or system APIs can finish after Swift cancellation -was requested. Result application therefore also validates work ID, request generation, focus -identity, content signature, settings continuity, and current field state. - -## Debounce and AX Publish Timing - -A global key event can arrive before the host application publishes its new value through -Accessibility. Reading immediately would build a request from pre-keystroke text. Cotabby combines -debounce with explicit focus refreshes and short host-publish polling when it knows the host is -catching up. - -The pipeline has a ceiling: if a host never publishes a detectable change, it eventually proceeds -through normal downstream guards instead of waiting forever. Freshness helpers avoid paying -duplicate synchronous AX walks when another pipeline stage just captured the same state. - -## Eligibility - -[SuggestionAvailabilityEvaluator.swift](../../Cotabby/Support/Suggestion/SuggestionAvailabilityEvaluator.swift) -contains pure gating rules. A prediction may be withheld because of: - -- Missing required permissions -- Global disable or temporary pause -- Per-application disable -- Unsupported or secure focus capability -- Selection or incompatible editing state -- Terminal policy -- Insufficient text signal -- Runtime or engine availability - -Gating is repeated at meaningful async boundaries. Passing eligibility before debounce does not -authorize applying a result after the user switches fields, changes settings, or selects text. - -## Corrections Before Model Generation - -Spelling correction is a native fast path evaluated before model generation. TypoGate determines -whether the current editing shape is eligible. CurrentWordSpellChecker uses NSSpellChecker, while -SymSpellCorrector supplies frequency-ranked multilingual candidates after its index is available. - -Depending on settings and the input event, the coordinator can: - -- Suppress model completion while the user is still building a likely typo. -- Present a correction as a green replace-the-word suggestion. -- Automatically replace a completed typo after Space. -- Continue to ordinary model generation when no correction applies. - -A correction session has different acceptance semantics from a continuation. It commits as one -atomic replacement rather than exposing partial word acceptance. - -## Request Construction - -[SuggestionRequestFactory.swift](../../Cotabby/Support/Suggestion/SuggestionRequestFactory.swift) builds an -immutable SuggestionRequest and the selected backend's developer-debug prompt payload. It: - -- Bounds the prefix according to the selected engine. -- Selects a language-aware prediction budget. -- Adds enabled extended, clipboard, visual, and surface context. -- Sanitizes and budgets optional context. -- Builds the base-model prompt used by llama and completion-style endpoints. -- Preserves structured request fields for the Apple prompt renderer. -- Assigns a request ID used across structured logs. - -The coordinator decides when to request. The factory decides what the request contains. Engines do -not reach back into live Accessibility state while generation is running. - -## Streaming and Final Results - -SuggestionGenerating exposes both a single-result method and a streaming method. Streaming engines -send cumulative, already-normalized partials. -[SuggestionStreamingState.swift](../../Cotabby/Support/Suggestion/SuggestionStreamingState.swift) -owns three pure bookkeeping rules: the newest pending partial wins, only one runloop drain is -scheduled at a time, and rendered text grows monotonically through StreamedGhostTextPolicy. The -coordinator still owns DispatchQueue scheduling, freshness checks, session creation, and overlay -side effects. - -A streamed partial can become a real active session, allowing acceptance before decoding finishes. -The final result remains authoritative. It may replace the partial, suppress it, or clear it if final -confidence and seam guards reject the completion. - -Streaming presentation is controlled by settings. Backend streaming support does not imply that -partials must always be painted. - -## Normalization and Display Guards - -SuggestionTextNormalizer applies backend-independent cleanup: - -- Removes control tokens and reasoning blocks. -- Strips echoed prompt scaffolding. -- Applies single-line or bounded multi-line policy. -- Rejects duplication of text already after the caret. -- Removes repeated prefix echoes. -- Reconciles leading whitespace. -- Rejects unsafe or empty insertions. - -The coordinator then applies display-time checks such as CompletionSeamGuard. That guard suppresses -junk punctuation runs and likely mid-word misspelling splices. Streaming uses the cheap pure part of -the guard; the authoritative final can run the full spelling-dependent verdict. - -## Active Sessions - -[SuggestionInteractionState.swift](../../Cotabby/Services/Suggestion/SuggestionInteractionState.swift) -owns: - -- The materialized focus context buffer -- The active suggestion session -- Consumed character count -- The sentinel indicating that Cotabby inserted text but AX has not published it yet - -[SuggestionSessionReconciler.swift](../../Cotabby/Support/Suggestion/SuggestionSessionReconciler.swift) -contains the pure rules for comparing a session with live editor state. It tolerates narrowly scoped -post-insertion AX lag while rejecting focus changes, altered trailing text, selection, undo, or -divergent typing. - -## Acceptance - -The word-accept key follows the configured word or phrase granularity. The full-accept key commits -the entire remaining tail. Acceptance validates: - -- Cotabby is still enabled. -- Input Monitoring remains granted. -- The focused field remains supported. -- A live session exists. -- The visible overlay matches the remaining session text. -- Current AX text still reconciles with the session anchor. - -Successful insertion advances or exhausts the session. Partial acceptance updates the visible tail -synchronously so a rapid second press cannot observe an overlay/session mismatch. - -After final acceptance, Cotabby starts speculative generation against the text the host is expected -to publish. A parallel publish check validates that guess. If the host publishes different content, -ordinary newer work supersedes the speculation. - -There is also a short gap between exhausting the visible tail and presenting a regenerated -continuation. [PostExhaustionAcceptanceState.swift](../../Cotabby/Support/Suggestion/PostExhaustionAcceptanceState.swift) -keeps Tab owned only during that bounded window, collapses repeated rapid presses into at most one -queued accept, and keys the timeout to the current arm generation. The coordinator owns the event-tap -and timer effects; the value owns only the transition rules. A timeout or teardown returns Tab to the -host, while a fresh continuation atomically consumes the queued accept. - -## Invariants - -- Only the current work ID may apply asynchronous results. -- Focus and content signatures are revalidated after awaits. -- An overlay is acceptable only when it matches the active session tail. -- Streamed partials are provisional; the final result is authoritative. -- A cancellation is not automatically a runtime failure. -- Stream drain and post-exhaustion acceptance rules live in pure state values; their timers, - scheduling, input interception, and presentation effects remain coordinator-owned. -- Native correction runs before model generation. -- AX lag tolerance is narrow and tied to a known Cotabby insertion. -- Pure policy stays in Support; mutable orchestration stays in the coordinator and service owners. - -## Failure-Oriented Reading - -- Suggestion never starts: availability evaluator, focus capability, settings snapshot. -- Suggestion uses text from before the keypress: host-publish polling and focus refresh timing. -- Old text appears after switching fields: work ID and focus/content signature guards. -- Partial appears then vanishes: final normalization, confidence, or seam suppression. -- Tab passes through despite visible text: overlay/session acceptance validation and input tap state. -- A rapid second Tab escapes after the tail is exhausted: post-exhaustion arm, queued accept, and - generation-keyed backstop. -- Accepted word repeats: post-insertion reconciliation and speculative-generation validation. -- Typo correction behaves like a continuation: session kind and correction acceptance path. -- Ghost tail jitters after partial acceptance: acceptance presentation and overlay advance logic. - -## Update This Guide When - -Update this document when a new pipeline state is introduced, a stale-result signature changes, -correction ordering moves, request construction gains a context source, streaming semantics change, -or acceptance/session reconciliation acquires a new invariant. diff --git a/.internal/interview-prep/README.md b/.internal/interview-prep/README.md deleted file mode 100644 index 322a1e52..00000000 --- a/.internal/interview-prep/README.md +++ /dev/null @@ -1,802 +0,0 @@ -# Cotabby Interview Study Path - -## Purpose - -This is an untimed, mastery-based study path for explaining Cotabby in a technical interview. It -does not repeat the detailed architecture guides. It tells you what to read, which concrete symbols -to trace, what decisions to understand, and what you must be able to reconstruct without notes. - -The objective is not to memorize the repository. The objective is to internalize the product as a -set of reliability boundaries: - -~~~text -process lifecycle - -> focused editor truth - -> global input intent - -> eligible immutable request - -> selected generation backend - -> normalized active session - -> non-activating presentation - -> validated insertion - -> host publication reconciliation -~~~ - -You are ready when you can enter the repository through a symptom or design question, name the -responsible owner, trace the data flow, explain the tradeoff, and identify the invariant that keeps -the user safe. - -## Source-of-Truth Order - -Use documentation in this order: - -1. [Root architecture map](../../ARCHITECTURE.md) for the whole-system mental model. -2. The relevant guide under [architecture](../architecture/) for the subsystem explanation. -3. The linked production source for current behavior. -4. Tests for executable examples and edge cases. - -[README.md](../../README.md) is the product-facing overview, while [AGENTS.md](../../AGENTS.md) is the -canonical coding-agent instruction file. Both are synchronized with the three-engine architecture, -but neither replaces the subsystem guides or production source for implementation-level questions. -When any prose conflicts with code and tests, verify the owner directly and fix the documentation. - -## How to Study - -For each section: - -1. Read the named architecture guide. -2. Open the production files in the stated order. -3. Locate the named symbols rather than scrolling randomly. -4. Draw the input, mutable owner, async boundary, output, and stale-result guard. -5. Complete the trace exercise without copying code. -6. Answer the checkpoint questions aloud. -7. Reopen the code only after you have committed to an answer. - -When explaining a decision, use this shape: - -~~~text -constraint - -> chosen boundary or mechanism - -> failure it prevents - -> cost or tradeoff - -> evidence in specific files - -> improvement you would consider -~~~ - -That format demonstrates engineering judgment. Merely describing the implementation demonstrates -code familiarity but not architectural understanding. - -## Mastery Area 1: Product and System Shape - -### Read - -- [Root architecture map](../../ARCHITECTURE.md) -- [Lifecycle and Composition](../architecture/lifecycle-and-composition.md) -- [Suggestion Pipeline](../architecture/suggestion-pipeline.md) - -### Open - -1. [CotabbyApp.swift](../../Cotabby/App/Core/CotabbyApp.swift) -2. [AppDelegate.swift](../../Cotabby/App/Core/AppDelegate.swift) -3. [CotabbyAppEnvironment.swift](../../Cotabby/App/Core/CotabbyAppEnvironment.swift) -4. [SuggestionCoordinator.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator.swift) -5. [SuggestionSettingsModel.swift](../../Cotabby/Models/Settings/SuggestionSettingsModel.swift) -6. [SuggestionSettingsData.swift](../../Cotabby/Models/Settings/SuggestionSettingsData.swift) - -### Find - -- CotabbyApp.body -- AppDelegate.applicationDidFinishLaunching -- AppDelegate.applicationWillTerminate -- CotabbyAppEnvironment.init -- SuggestionCoordinator dependencies, internal state, and the small UI-observed published surface -- SuggestionSettingsModel.domainSettings and snapshot - -### Understand - -Cotabby is not primarily a text-generation application. It is a cross-process editor integration -whose model call sits in the middle of a longer state machine. Reliability depends on focus truth, -event integrity, stale-work rejection, overlay ownership, insertion correctness, and host -reconciliation. - -CotabbyApp declares SwiftUI scenes. CotabbyAppEnvironment constructs one shared dependency graph. -AppDelegate controls when app-scoped side effects begin and end. This prevents duplicate AX polling, -event taps, runtime managers, panels, and settings models. - -Settings have one durable owner. SwiftUI continues to bind the model's individual published -properties, `domainSettings` projects them into cohesive product areas, the snapshot freezes the -behavior subset for async work, and the store preserves flat UserDefaults keys. - -### Trace Exercise - -Draw process startup from the App entry point through environment construction to these services: - -- FocusTracker -- InputMonitor -- SuggestionCoordinator -- VisualContextCoordinator -- LlamaRuntimeManager -- InlineCommandCoordinator - -For each one, distinguish construction from startup. Then reverse the drawing for process -termination. - -### Checkpoints - -- Why is constructing a service different from starting it? -- Which subscriptions belong to CotabbyAppEnvironment, and which belong to AppDelegate? -- Why would constructing FocusTracker or InputMonitor inside a SwiftUI view be dangerous? -- Why does the XCTest host skip production startup? -- What resources must be stopped before native runtime shutdown? -- Give a one-minute description of Cotabby without leading with llama.cpp. - -## Mastery Area 2: Focus as Eventually Consistent State - -### Read - -- [Focus and Accessibility](../architecture/focus-and-accessibility.md) -- The privacy boundary in [Context, Privacy, and Permissions](../architecture/context-privacy-and-permissions.md) - -### Open - -1. [FocusTracker.swift](../../Cotabby/Services/Focus/FocusTracker.swift) -2. [FocusSnapshotResolver.swift](../../Cotabby/Services/Focus/FocusSnapshotResolver.swift) -3. [FocusModels.swift](../../Cotabby/Models/Focus/FocusModels.swift) -4. [AXTextGeometryResolver.swift](../../Cotabby/Services/Focus/AXTextGeometryResolver.swift) -5. [AXHelper.swift](../../Cotabby/Support/Accessibility/AXHelper.swift) - -### Find - -- FocusTracker.start, refreshNow, performCaptureAndPublish, and resolveChromiumFocusFallback -- FocusSnapshotResolver.resolveSnapshot, resolveCandidate, boundedContextWindow, and candidateSnapshot -- FocusCapability and FocusedInputSnapshot -- CaretGeometryQuality -- AXTextGeometryResolver.resolveCaretRect and its range, marker, character, static-run, and field - fallback branches - -### Understand - -AX is synchronous cross-process IPC. Different applications expose different roles, selection -representations, text ranges, and geometry. A FocusSnapshot is therefore a normalized observation, -not permanent truth. - -Polling is authoritative because AXObserver coverage and ordering are inconsistent. Explicit refresh -is still a complete capture, not trust in a notification payload. Adaptive backoff reduces idle cost. -Chromium accessibility priming and hit testing recover browser cases that the system-wide focused -element query misses. - -Identity is deliberately layered. Element identifiers can be recycled, so focusChangeSequence, -process identity, content signatures, and selection are used according to the downstream invariant. - -### Trace Exercise - -Trace a focused Gmail editor from FocusTracker.performCaptureAndPublish through -FocusSnapshotResolver.resolveSnapshot. Record: - -- How the owning process is determined -- How editable candidates are ranked -- Where text is bounded -- Where secure capability is decided -- How caret geometry receives a quality -- Which values become FocusedInputSnapshot -- What causes a new focusChangeSequence - -Repeat conceptually for a native NSTextView and note which compatibility branches disappear. - -### Checkpoints - -- Why is NSWorkspace.frontmostApplication insufficient? -- Why is an editable AX role insufficient? -- Why bound text before publishing the snapshot rather than only in the prompt renderer? -- What makes exact, derived, estimated, and layoutEstimated geometry different? -- Why does presentation care about geometry quality? -- What can go wrong if a cache is keyed only by elementIdentifier? -- Why are AX calls kept on MainActor despite their latency risk? -- What is the current secure-field acquisition limitation? - -## Mastery Area 3: Global Input Without Stealing Keystrokes - -### Read - -- [Input and Insertion](../architecture/input-and-insertion.md) -- Input handling in [Suggestion Pipeline](../architecture/suggestion-pipeline.md) - -### Open - -1. [InputMonitor.swift](../../Cotabby/Services/Input/InputMonitor.swift) -2. [InputSuppressionController.swift](../../Cotabby/Services/Input/InputSuppressionController.swift) -3. [KeyboardInputSourceMonitor.swift](../../Cotabby/Services/Input/KeyboardInputSourceMonitor.swift) -4. [InlineCommandCoordinator.swift](../../Cotabby/App/Coordinators/InlineCommandCoordinator.swift) - -### Find - -- InputMonitor.start and refresh -- installObserverTapIfNeeded -- installAcceptTapIfNeeded -- installToggleTapIfNeeded -- updateAcceptTapState -- handleObserverKeyDown -- handleAcceptKeyDown -- acceptanceKind -- InputMonitorAcceptTapDecision - -### Understand - -The steady observer is listen-only and cannot consume user input. A separate default tap exists only -while a visible suggestion or inline-command capture needs interception. Even then, a matching key is -consumed only after the current owner returns success. Stale or declined ownership passes the -original event through. - -The global toggle has independent lifetime because it must work without a visible suggestion. -InlineCommandCoordinator arbitrates the one capture slot between emoji and macros. - -Synthetic insertion events are tagged and covered by bounded suppression so Cotabby does not treat -its own writes as new user typing. Suppression must expire quickly enough that real typing is never -hidden. - -### Trace Exercise - -Trace one configured word-accept key twice: - -1. A valid visible suggestion exists and acceptance succeeds. -2. The consuming tap still exists but the session became stale before the key arrived. - -Show exactly why the first event is swallowed and the second reaches the host application. - -Then trace one Unicode event posted by SuggestionInserter and explain why neither the listen-only -observer nor the consuming accept tap should treat it as user intent. - -### Checkpoints - -- Why not install one permanently consuming event tap? -- What does fail-open mean here? -- Why does key recognition read current settings at event time? -- Why does the accept tap linger briefly after the overlay hides? -- Why does a synthetic source marker exist in addition to a suppression count? -- How can emoji use the accept key without racing suggestion acceptance? -- What should happen if Input Monitoring permission is revoked while a suggestion is visible? - -## Mastery Area 4: Scheduling and Building a Request - -### Read - -- [Suggestion Pipeline](../architecture/suggestion-pipeline.md) -- [Context, Privacy, and Permissions](../architecture/context-privacy-and-permissions.md) - -### Open - -1. [SuggestionCoordinator+Input.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator+Input.swift) -2. [SuggestionCoordinator+Prediction.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator+Prediction.swift) -3. [SuggestionWorkController.swift](../../Cotabby/Services/Suggestion/SuggestionWorkController.swift) -4. [SuggestionAvailabilityEvaluator.swift](../../Cotabby/Support/Suggestion/SuggestionAvailabilityEvaluator.swift) -5. [SuggestionRequestFactory.swift](../../Cotabby/Support/Suggestion/SuggestionRequestFactory.swift) -6. [SuggestionRequest.swift](../../Cotabby/Models/Suggestion/SuggestionRequest.swift) - -### Find - -- handleFocusSnapshotChange -- handleInputEvent -- schedulePredictionAfterHostPublishDelay -- pollForHostPublish -- schedulePrediction -- generateFromCurrentFocus -- dispatchGeneration -- SuggestionWorkController.replaceDebouncedWork, replaceGenerationWork, isCurrent, and cancelAll -- SuggestionAvailabilityEvaluator.disabledReason -- SuggestionRequestFactory.buildRequest - -### Understand - -A global keydown often arrives before the host publishes its new AX value. The coordinator therefore -does not immediately assume the snapshot contains the typed character. It debounces, requests fresh -focus state, and performs bounded host-publication polling. - -Cancellation is advisory. Native or system work may complete after cancellation, so every unit of -prediction work also carries a monotonically increasing work ID. Result application validates that -ID plus focus, content, session, and settings continuity. - -SuggestionRequestFactory separates the question of what to request from when to request. It receives -already captured values, applies engine-specific bounds and optional-context budgets, and returns an -immutable SuggestionRequest with a request ID. - -### Trace Exercise - -Trace the typed character a from InputMonitor through: - -- host-publication delay -- availability evaluation -- work replacement -- fresh focus materialization -- correction gate -- clipboard relevance -- visual excerpt lookup -- request construction -- engine dispatch - -At each await, write the condition that could make the work stale. - -### Checkpoints - -- Why are cancellation and work identity both required? -- Why gate eligibility more than once? -- Why does an engine receive an immutable request rather than a focus service? -- Which optional contexts can enter a request? -- Which layer owns context acquisition, and which owns prompt budgeting? -- Why does the caret prefix receive priority over optional context? -- Why does every request need a correlation ID? -- What is speculative post-acceptance generation, and how is an incorrect speculation rejected? - -## Mastery Area 5: Streaming, Sessions, and Reconciliation - -### Read - -- Streaming, normalization, and sessions in [Suggestion Pipeline](../architecture/suggestion-pipeline.md) -- Presentation stability in [Presentation and Sibling Features](../architecture/presentation-and-sibling-features.md) - -### Open - -1. [SuggestionCoordinator+Prediction.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator+Prediction.swift) -2. [SuggestionInteractionState.swift](../../Cotabby/Services/Suggestion/SuggestionInteractionState.swift) -3. [SuggestionSessionReconciler.swift](../../Cotabby/Support/Suggestion/SuggestionSessionReconciler.swift) -4. [SuggestionTextNormalizer.swift](../../Cotabby/Support/Suggestion/SuggestionTextNormalizer.swift) -5. [CompletionSeamGuard.swift](../../Cotabby/Support/Suggestion/CompletionSeamGuard.swift) -6. [StreamedGhostTextPolicy.swift](../../Cotabby/Support/Suggestion/StreamedGhostTextPolicy.swift) -7. [SuggestionStreamingState.swift](../../Cotabby/Support/Suggestion/SuggestionStreamingState.swift) - -### Find - -- queueStreamedPartial, drainStreamedPartial, and applyStreamedPartial -- SuggestionStreamingState.beginGeneration, enqueue, drain, canRender, and clearSession -- apply(result:workID:) -- SuggestionInteractionState.startSession and reconcileActiveSession -- advanceIfTypedCharactersMatch -- SuggestionSessionReconciler.reconcile -- nextAcceptanceChunk and nextAcceptancePhrase -- SuggestionTextNormalizer.normalizeDetailed - -### Understand - -Streamed partials are cumulative and provisional. SuggestionStreamingState makes pending partials -latest-wins, permits only one scheduled drain per runloop window, and remembers the monotonic rendered -prefix. The coordinator still owns scheduling, freshness checks, session mutation, and AppKit. A -partial can become a real active session so the user can accept before decoding ends, but the final -result can replace or suppress it. - -SuggestionInteractionState owns mutable session facts. SuggestionSessionReconciler owns pure -comparison rules. This split lets acceptance, type-through, trailing-text checks, CJK segmentation, -and known post-insertion lag be tested without running AX or AppKit. - -### Trace Exercise - -Start with the visible suggestion: - -~~~text - meeting tomorrow at 10 -~~~ - -Trace these independent cases: - -- The user types the exact leading space and m -- The user types a divergent character -- The user accepts one word -- The user accepts the full tail -- AX briefly reports the pre-insertion value -- The final model result is rejected after a partial was displayed - -Identify which owner changes session state and which pure rule decides the transition. - -### Checkpoints - -- Why can a streamed partial become accept-ready? -- Why must the final result remain authoritative? -- What is the invariant between visible overlay text and remaining session text? -- Why is post-insertion AX tolerance represented by a narrow sentinel? -- How does exact type-through avoid unnecessary regeneration? -- Why are corrections represented as a different session kind? -- Where do language-specific acceptance rules belong? - -## Mastery Area 6: Presentation and Insertion - -### Read - -- [Presentation and Sibling Features](../architecture/presentation-and-sibling-features.md) -- [Input and Insertion](../architecture/input-and-insertion.md) - -### Open - -1. [SuggestionOverlayPresenter.swift](../../Cotabby/Services/Suggestion/SuggestionOverlayPresenter.swift) -2. [OverlayController.swift](../../Cotabby/Services/Presentation/OverlayController.swift) -3. [CompletionRenderModePolicy.swift](../../Cotabby/Support/Presentation/CompletionRenderModePolicy.swift) -4. [SuggestionOverlayStabilityGate.swift](../../Cotabby/Support/Presentation/SuggestionOverlayStabilityGate.swift) -5. [SuggestionCoordinator+Acceptance.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator+Acceptance.swift) -6. [SuggestionInserter.swift](../../Cotabby/Services/Suggestion/SuggestionInserter.swift) -7. [InsertionSafetyGate.swift](../../Cotabby/Support/Suggestion/InsertionSafetyGate.swift) -8. [InsertionStrategySelector.swift](../../Cotabby/Support/Suggestion/InsertionStrategySelector.swift) -9. [PostExhaustionAcceptanceState.swift](../../Cotabby/Support/Suggestion/PostExhaustionAcceptanceState.swift) - -### Find - -- SuggestionOverlayPresenter.present -- OverlayController.showSuggestion, showInline, showMirror, and advanceInline -- CompletionRenderModePolicy.mode -- acceptCurrentSuggestion, acceptEntireSuggestion, and acceptEnabledSuggestion -- presentAdvancedOverlay and schedulePostInsertionRefresh -- armPostExhaustionAcceptance and flushQueuedPostExhaustionAcceptIfNeeded -- SuggestionInserter.insert, replace, insertViaPaste, and pressPasteMenuItem - -### Understand - -The panel is borderless, non-activating, mouse-ignoring, and space/full-screen compatible. AppKit owns -window behavior; SwiftUI describes its contents. - -Automatic mode paints inline only with exact or derived geometry and a safe end-of-line seam. -Estimated, layout-estimated, and mid-line cases use a mirror card. This avoids visually pretending -that approximate geometry identifies an exact glyph position. - -Short ordinary insertions use Unicode CGEvents. A composing IME uses paste because a Unicode event -can re-enter composition. Optional long/multiline paste snapshots every pasteboard representation, -tries the target application's AX Paste menu item, falls back to Command-V, and restores only if the -clipboard has not changed. - -### Trace Exercise - -Trace a partial word acceptance from acceptCurrentSuggestion through session preparation, insertion, -overlay advance, post-insertion sentinel, focus refresh, and session reconciliation. - -Repeat for: - -- A Japanese IME-active field -- A Chrome field where synthetic Command-V fails -- A correction that replaces a typo -- A user clipboard change during the restore delay - -### Checkpoints - -- Why does OverlayController own NSPanel instead of SuggestionCoordinator? -- Why is layoutEstimated still a mirror-card quality? -- Why can the overlay advance without waiting for fresh AX geometry? -- What protects the user's clipboard during overlapping paste insertions? -- Why is posting a CGEvent not proof that insertion succeeded? -- Why must acceptance validate both the session and visible overlay? -- Why can rapid Tab queue only one unseen accept while an exhausted tail regenerates? -- How does the generation-keyed backstop guarantee Tab ownership returns to the host? -- What is the safest behavior when any acceptance precondition fails? - -## Mastery Area 7: The Three Generation Backends - -### Read - -- [Inference and Prompting](../architecture/inference-and-prompting.md) -- Engine privacy in [Context, Privacy, and Permissions](../architecture/context-privacy-and-permissions.md) - -### Open - -1. [SuggestionEngineRouter.swift](../../Cotabby/Services/Runtime/SuggestionEngineRouter.swift) -2. [FoundationModelSuggestionEngine.swift](../../Cotabby/Services/Runtime/FoundationModelSuggestionEngine.swift) -3. [LlamaSuggestionEngine.swift](../../Cotabby/Services/Runtime/LlamaSuggestionEngine.swift) -4. [LlamaRuntimeManager.swift](../../Cotabby/Services/Runtime/LlamaRuntimeManager.swift) -5. [LlamaRuntimeCore.swift](../../Cotabby/Services/Runtime/LlamaRuntimeCore.swift) -6. [OpenAICompatibleSuggestionEngine.swift](../../Cotabby/Services/Runtime/OpenAICompatibleSuggestionEngine.swift) -7. [OpenAICompatibleAPIClient.swift](../../Cotabby/Services/Runtime/OpenAICompatibleAPIClient.swift) -8. [BaseCompletionPromptRenderer.swift](../../Cotabby/Support/Prompting/BaseCompletionPromptRenderer.swift) -9. [FoundationModelPromptRenderer.swift](../../Cotabby/Support/Prompting/FoundationModelPromptRenderer.swift) - -### Find - -- SuggestionEngineRouter.generateSuggestion, prewarm, resetCachedGenerationContext, and - generateOpenSourceFallback -- FoundationModelSuggestionEngine.ensureSession -- LlamaSuggestionEngine.generateSuggestion and resetCachedGenerationContext -- LlamaRuntimeManager.prepare, generate, stop, stopAndWait, and shutdownSync -- LlamaRuntimeCore.prepare, generate, preparedPrompt, obtainAutocompleteSequence, - resetPromptCache, and shutdown -- OpenAICompatibleSuggestionEngine.generateSuggestion and prewarm -- BaseCompletionPromptRenderer.prompt -- FoundationModelPromptRenderer.sessionInstructions and prompt - -### Understand - -The router provides one generation contract while preserving backend-specific lifecycle: - -- Apple uses an instruction channel and one-use compatible prewarmed session. -- Llama uses a base completion prompt, in-process native pointers, prompt token/KV reuse, token - streaming, and explicit shutdown. -- An OpenAI-compatible endpoint uses completion or chat request transport and SSE parsing. It may be - loopback, LAN, or public HTTPS. - -LlamaRuntimeManager is MainActor and publishes user-facing state. LlamaRuntimeCore is a nonisolated, -lock/condition-protected class that owns native correctness. It is not a Swift actor. The -autocomplete lock serializes prompt-cache/decode state; lifecycle coordination protects load, -active operations, abort, and shutdown. - -### Trace Exercise - -Take one SuggestionRequest and trace it separately through all three engines. For each path identify: - -- Prompt shape -- Prewarm behavior -- Streaming mechanism -- Cancellation mechanism -- Cache or session state -- Output normalization -- Error classification -- Privacy boundary -- Cleanup behavior - -Then trace an engine switch from Open Source to endpoint and explain why the llama runtime is stopped. - -### Checkpoints - -- Why does Apple have a narrow fallback to llama rather than universal silent fallback? -- Why are base GGUF prompts continuation-shaped? -- Why does the caret prefix come last? -- Why is LlamaRuntimeCore not simply MainActor-isolated? -- Why use explicit locking instead of casually wrapping native pointers in Task calls? -- What is safe KV-cache reuse? -- Why broadcast context reset to every backend? -- What data can leave the Mac in endpoint mode? -- Why reject insecure public HTTP but allow loopback HTTP? - -## Mastery Area 8: Context, Permissions, and Honest Privacy - -### Read - -- [Context, Privacy, and Permissions](../architecture/context-privacy-and-permissions.md) - -### Open - -1. [PermissionManager.swift](../../Cotabby/Services/Permission/PermissionManager.swift) -2. [PermissionGuidanceController.swift](../../Cotabby/Services/Permission/PermissionGuidanceController.swift) -3. [ClipboardContextProvider.swift](../../Cotabby/Services/Context/ClipboardContextProvider.swift) -4. [ClipboardRelevanceFilter.swift](../../Cotabby/Support/Context/ClipboardRelevanceFilter.swift) -5. [VisualContextCoordinator.swift](../../Cotabby/Services/Visual/VisualContextCoordinator.swift) -6. [WindowScreenshotService.swift](../../Cotabby/Services/Visual/WindowScreenshotService.swift) -7. [ScreenTextExtractor.swift](../../Cotabby/Services/Visual/ScreenTextExtractor.swift) -8. [OCRTextHygiene.swift](../../Cotabby/Support/Context/OCRTextHygiene.swift) -9. [PromptContextSanitizer.swift](../../Cotabby/Support/Context/PromptContextSanitizer.swift) -10. [OpenAICompatibleEndpointModels.swift](../../Cotabby/Models/Runtime/OpenAICompatibleEndpointModels.swift) - -### Find - -- PermissionManager.refresh and requiredPermissionsGranted -- ClipboardRelevanceFilter.filter -- VisualContextCoordinator.startSessionIfNeeded, launchSession, and applyExcerpt -- ScreenshotContextGenerator.generateContext and finishedExcerpt -- OCRTextHygiene.clean -- OpenAICompatibleEndpointConfiguration validation and privacyWarning -- SuggestionAvailabilityEvaluator.shouldCaptureVisualContext - -### Understand - -Accessibility and Input Monitoring are required for core autocomplete. Screen Recording is optional. -Visual context is a field-scoped screenshot-to-Vision-OCR pipeline. OCR confidence and hygiene remove -noise; no model summarization occurs; only a bounded sanitized excerpt can reach a prompt. - -Clipboard context is read at request time, relevance-filtered, distilled, and not retained as a -history. User-authored context, surface metadata, and visual context have independent budgets. - -Privacy claims must distinguish on-device Apple/llama generation from a configured endpoint. They -must also describe the current secure-field limitation honestly: generation and insertion are -blocked, but a bounded FocusedInputSnapshot is still created and visual capture can run because the -visual eligibility gate ignores capability. - -### Trace Exercise - -Build a context inventory for one request. For each field record: - -- Acquisition owner -- Permission -- In-memory lifetime -- First bound -- Prompt budget -- Whether it can be logged in debug mode -- Whether it leaves the Mac under each engine - -Then trace a secure field far enough to show where assistance stops and where acquisition currently -does not. - -### Checkpoints - -- Why is Screen Recording optional? -- Why is visual context scoped to a field instead of regenerated on every key? -- Why is raw OCR cleaned rather than summarized by another model? -- Why does relevance filtering matter for clipboard context? -- What exactly does local-first mean in the current product? -- Where are endpoint credentials stored? -- Which privacy claim would currently be false? -- How would you move toward a true no-secure-field-acquisition invariant? - -## Mastery Area 9: Observability, Tests, and Failure Diagnosis - -### Read - -- Debugging and validation in [Root architecture map](../../ARCHITECTURE.md) -- Failure-oriented sections in every architecture guide - -### Open - -1. [CotabbyDebugOptions.swift](../../Cotabby/Support/Logging/CotabbyDebugOptions.swift) -2. [RequestID.swift](../../Cotabby/Support/Logging/RequestID.swift) -3. [SuggestionDebugLogger.swift](../../Cotabby/Services/Suggestion/SuggestionDebugLogger.swift) -4. [FileLogHandler.swift](../../Cotabby/Support/Logging/FileLogHandler.swift) -5. [LLMIOFileHandler.swift](../../Cotabby/Support/Logging/LLMIOFileHandler.swift) -6. [CotabbyTests](../../CotabbyTests) -7. [build workflow](../../.github/workflows/build.yml) -8. [test workflow](../../.github/workflows/tests.yml) -9. [XcodeGen workflow](../../.github/workflows/xcodegen.yml) -10. [lint workflow](../../.github/workflows/lint.yml) - -### Understand - -A request ID joins coordinator stages, backend selection, generation, performance, acceptance, and -debug LLM I/O. The always-on path uses unified logging. Explicit -cotabby-debug enables local JSONL, -full prompts/completions, AX dumps, and visual capture artifacts. - -Tests are strongest around deterministic Support and Models rules. Stateful owners expose narrow -protocols and fakes so work identity, session transitions, trigger machines, prompt rendering, and -layout policy can run without global permissions. - -XcodeGen makes project.yml the project source of truth. CI intentionally treats a regenerated -project diff as a failure. - -### Trace Exercise - -For each symptom, identify the first log category, correlation key, production file, and pure test -surface: - -- A suggestion from the previous field appears -- Tab is swallowed without insertion -- A Chrome caret is one line too low -- Llama remains resident after switching engines -- Visual context belongs to the previous field -- A final result suppresses a streamed partial - -### Checkpoints - -- Why is a request ID more useful than chronological logs alone? -- Which debug artifacts can contain private text? -- What should be unit tested without launching the app? -- What requires a real host-application compatibility test? -- Why are build, tests, lint, and XcodeGen separate CI checks? -- What would you instrument before trying to fix an intermittent editor bug? - -## The Six Golden Traces - -You should be able to draw these from memory and name the files at every arrow. - -### Trace 1: Process Startup - -~~~text -CotabbyApp - -> AppDelegate.init - -> CotabbyAppEnvironment.init - -> applicationDidFinishLaunching - -> runtime / focus / input / suggestion / inline command startup -~~~ - -### Trace 2: Focus Acquisition - -~~~text -FocusTracker poll - -> focused AX element or browser fallback - -> FocusSnapshotResolver - -> bounded FocusedInputSnapshot + capability - -> FocusTrackingModel publication - -> SuggestionCoordinator focus reaction -~~~ - -### Trace 3: Typed Character to Streamed Ghost Text - -~~~text -listen-only input event - -> host-publication refresh - -> availability - -> work ID - -> request factory - -> engine router - -> normalized cumulative partial - -> active session - -> overlay presenter -~~~ - -### Trace 4: Word Acceptance - -~~~text -conditional consuming tap - -> session + overlay validation - -> next acceptance chunk - -> SuggestionInserter - -> synchronous tail advance - -> post-insertion AX sentinel - -> fresh snapshot reconciliation - -> bounded post-exhaustion ownership if the tail ended -~~~ - -### Trace 5: Field Switch During Generation - -~~~text -new focus sequence - -> cancel old tasks - -> increment work identity - -> reset backend context - -> clear old session / overlay - -> late old result rejected by current-work and focus/content checks -~~~ - -### Trace 6: Engine Switch - -~~~text -settings/profile selection - -> cancel prediction - -> reset cached generation context - -> router selects backend - -> AppDelegate starts or stops llama runtime - -> selected backend prewarm -~~~ - -## Interview Readiness Checklist - -### Product - -- Explain Cotabby in one minute without reducing it to an LLM wrapper. -- State the required and optional permissions. -- Explain local-first without falsely claiming every engine is offline. -- Name the supported presentation and acceptance modes. - -### Ownership - -- Identify the composition root and lifecycle owner. -- Explain why views do not construct process-wide services. -- Distinguish coordinator orchestration, service side effects, model state, and pure Support rules. -- Explain why both environment and AppDelegate retain subscriptions. - -### Reliability - -- Explain focus eventual consistency. -- Explain cancellation plus work identity. -- Explain focus/content/session revalidation after awaits. -- Explain fail-open input consumption. -- Explain post-insertion reconciliation. -- Explain streamed partial versus authoritative final. -- Explain latest-wins stream draining and the bounded post-exhaustion rapid-accept window. - -### macOS - -- Explain the roles of Accessibility, Input Monitoring, and Screen Recording. -- Explain why AppKit panels are needed next to SwiftUI. -- Explain browser/Electron compatibility work. -- Explain IME-safe insertion and clipboard restoration. -- Explain TCC identity and why Cotabby Dev is separate. - -### Inference - -- Compare Apple, llama, and endpoint prompt/stream/lifecycle behavior. -- Explain manager versus core ownership. -- Explain safe prompt/KV reuse. -- Explain backend-independent normalization. -- Explain when data can leave the machine. - -### Critical Judgment - -- Identify current architecture strengths. -- Identify current complexity or debt without being dismissive. -- Explain the secure-field acquisition limitation honestly. -- Propose a measured improvement rather than a rewrite. -- Translate the invariants into a reliability plan for HyperWrite. - -## Final Active-Recall Drill - -Close the codebase and answer: - -1. What are the three most dangerous races in Cotabby? -2. What are the three ways Cotabby could accidentally interfere with another application? -3. Which owner is responsible for preventing each one? -4. Where does untrusted or stale information become a stable domain value? -5. Which state is app-scoped, field-scoped, request-scoped, session-scoped, and token-stream-scoped? -6. What happens when the host never publishes the insertion Cotabby expected? -7. What happens when the user switches engines during decode? -8. Which work stays on MainActor and which work must leave it? -9. Which claims are product promises, and which are current implementation limitations? -10. Which Cotabby patterns should transfer to HyperWrite, and which should be reconsidered? - -If an answer is vague, return to the named trace rather than rereading the repository from the -beginning. diff --git a/.internal/interview-prep/hyperwrite-reliability-translation.md b/.internal/interview-prep/hyperwrite-reliability-translation.md deleted file mode 100644 index e11bc684..00000000 --- a/.internal/interview-prep/hyperwrite-reliability-translation.md +++ /dev/null @@ -1,878 +0,0 @@ -# Translating Cotabby into a Reliable HyperWrite Mac Alpha - -## Purpose - -This document prepares you to discuss how Cotabby's lessons apply to the HyperWrite Mac prototype. -It is not a claim about HyperWrite's current implementation; you have not inspected that prototype -yet. It separates: - -- Questions that must be answered during the technical session -- Reliability invariants that apply to almost any cross-application Mac writing assistant -- Cotabby patterns worth transferring -- Cotabby-specific decisions that should not be copied blindly -- A practical sequence for turning a prototype into an alpha users can trust - -The goal is to show Josh that you can productize an existing prototype without prematurely rewriting -it or treating model integration as the whole problem. - -## The Core Interview Thesis - -A strong opening position is: - -> I would treat HyperWrite Mac as a cross-application interaction state machine, not just an API -> client with an overlay. The user has to trust that we understand the current editor, never show a -> stale suggestion, never steal an unrelated key, and insert exactly what was accepted. I would first -> instrument the prototype and define those invariants, then harden focus, input, session, insertion, -> and failure recovery around a declared compatibility matrix. - -That communicates three things: - -1. You understand where Mac-wide autocomplete fails in production. -2. You will learn from the prototype before replacing it. -3. You define reliability in observable behavior rather than general confidence. - -## Reliability Is a Product Contract - -For this product, reliable should mean: - -### Input integrity - -- HyperWrite never consumes a key it did not successfully handle. -- Synthetic insertion never re-enters the suggestion pipeline as user typing. -- Shortcut changes take effect predictably. -- An active IME does not cause acceptance to disappear into composition. - -### Target correctness - -- A suggestion belongs to one focused field and one text seam. -- Switching applications, fields, selections, or documents invalidates incompatible work. -- A late network or model response cannot appear in a newer editor state. - -### Session correctness - -- Visible suggestion text and the active remaining tail agree. -- Type-through advances only on exact matching characters. -- Partial acceptance inserts one deterministic chunk. -- A final stream result cannot silently corrupt an already accepted session. - -### Insertion correctness - -- Accepted text is committed through a strategy appropriate to the host and input method. -- The original acceptance key is consumed only when the write path succeeds. -- The host's subsequent state is reconciled; posting an event is not considered proof. -- Clipboard fallbacks preserve user data and respect newer clipboard changes. - -### Presentation correctness - -- HyperWrite UI never steals focus from the assisted editor. -- Approximate caret geometry is not presented as exact. -- The overlay stays within the correct screen/Space and degrades when positioning is uncertain. -- Hiding or advancing presentation cannot leave an accept-ready invisible session. - -### Availability and recovery - -- Missing permissions, network, authentication, model availability, sleep/wake, or host exit produce - explicit recoverable states. -- Cancellation is normal lifecycle, not an alarming error. -- Failed prewarm or optional context does not necessarily disable the core loop. -- Shutdown cannot race active native or network work. - -### Privacy and security - -- Every acquired context source has a permission, lifetime, bound, transport scope, and disclosure. -- Secure fields are rejected at the earliest practical acquisition boundary. -- Credentials use platform secret storage. -- Debug artifacts containing user text require an explicit mode and retention policy. - -### Observability - -- One suggestion can be traced from focus to request to stream to presentation to acceptance. -- Failures are classified by stage and reason rather than flattened into “did not work.” -- Metrics distinguish unsupported hosts from product regressions. - -## What Is Known and What Is Not - -Known from Josh's message: - -- There is a current HyperWrite Mac prototype. -- The objective is a strong Mac alpha. -- The technical session is intended to examine architecture and approach. -- Reliability and longer-term engineering fit are part of the evaluation. - -Unknown until the session: - -- Whether the app is Swift/AppKit/SwiftUI, Electron, Catalyst, or mixed -- How it discovers focused editors -- Whether it uses AX polling, notifications, event taps, or application-specific integration -- Which keys it observes or consumes -- Whether generation is local, hosted, hybrid, or streamed -- How request identity and cancellation work -- How it positions and owns its overlay -- How it inserts accepted text -- Which applications and languages are in alpha scope -- What telemetry or diagnostics already exist -- How it is signed, distributed, updated, and granted TCC permissions -- Which prototype failures are already known - -Do not fill these gaps with assumptions. Use them to demonstrate disciplined discovery. - -## Questions to Ask During the Technical Session - -### Product contract - -- What is the smallest user journey that must feel excellent in the alpha? -- Is the primary interaction inline autocomplete, rewrite commands, chat, or several modes? -- What applications are explicitly in scope? -- Are browsers, Electron editors, native fields, Office apps, and IDEs equally important? -- Are multiline, mid-line, and selected-text operations required? -- What are the intended acceptance and dismissal gestures? -- What does Josh mean by “strong alpha”: internal dogfood, invited users, or public release? -- Which current prototype behaviors are most embarrassing or unreliable? - -### Focus and editor state - -- How is the focused editable element discovered today? -- What representation of text, selection, and caret geometry is treated as authoritative? -- How are browser iframes and custom editors handled? -- Are secure/read-only/terminal fields classified? -- Is text bounded before it enters application state? -- What identifies the same field across AX element replacement? -- Is there a known compatibility matrix? - -### Input - -- Does the prototype use CGEvent taps, NSEvent global monitors, local monitors, or another mechanism? -- Is the tap listen-only or capable of consuming events? -- Under what exact condition is an acceptance key swallowed? -- How are synthetic writes distinguished from physical input? -- How do shortcut changes and modifier state interact with active capture? -- What happens when Input Monitoring permission is revoked? - -### Request and generation - -- Where is an immutable request assembled? -- Which context sources can be included? -- Is generation streamed, and are partials cumulative or deltas? -- What backend owns authentication, retries, timeout, and cancellation? -- Can responses arrive out of order? -- Is there request or session identity across client and server logs? -- What quality cleanup happens before display? -- What is the desired behavior when the backend is slow or offline? - -### Presentation - -- Is the overlay an NSPanel, SwiftUI scene, Electron window, or host-integrated view? -- Can it become key or steal focus? -- How is caret geometry obtained and classified? -- What happens when exact geometry is unavailable? -- How are multiple displays, Spaces, full-screen apps, RTL, and multiline text handled? -- Can a visible partial be accepted before the final response? - -### Insertion - -- Does acceptance use Unicode events, AX value mutation, paste, menu commands, or per-app strategies? -- How is the host result verified? -- How are IMEs handled? -- What happens if the host ignores or transforms the inserted text? -- If the clipboard is used, how are all representations restored? -- Are replacements and forward continuations represented differently? - -### Privacy - -- Which user text and screen context leaves the machine? -- What is stored by the client and server? -- Are screenshots or OCR involved? -- How are secure fields excluded? -- Where are credentials stored? -- What does debug logging contain and how long is it retained? -- Which privacy claims are already public? - -### Distribution and operations - -- What macOS versions and hardware are supported? -- Is the app sandboxed? Why or why not? -- How are signing, notarization, entitlements, and updates handled? -- Are development and production TCC identities separate? -- Is there a crash-reporting or support-diagnostics path? -- How are releases rolled back? - -### Code and team - -- Which parts of the prototype are considered sound and should be preserved? -- Where does Josh expect architectural change? -- What tests exist? -- What build/release automation exists? -- Who decides product tradeoffs during the trial? -- What access, designs, backend contracts, and user feedback will be available? - -## How to Inspect the Prototype Live - -Ask Josh to demonstrate one successful suggestion and one known failure. Trace both through the same -questions: - -~~~text -What editor state did the app observe? - -> What event triggered work? - -> What request identity was created? - -> What context was sent? - -> What backend operation ran? - -> What partial/final output returned? - -> What state became accept-ready? - -> What window displayed it? - -> What consumed the acceptance key? - -> What inserted text? - -> How was host success verified? - -> Which logs prove the path? -~~~ - -Do not start with “I would rewrite this.” Start with: - -- Where is the state owned? -- Which invariant is implicit? -- Which boundary lacks identity or observability? -- Is the observed failure deterministic, host-specific, or timing-specific? -- Can the current component be wrapped behind a reliable contract? - -## Provisional Risk Register - -These risks should be validated, not assumed. - -| Risk | User-visible failure | First evidence to seek | Cotabby lesson | -| --- | --- | --- | --- | -| Stale focus | Suggestion appears in the wrong field | Focus/session identifiers in logs | focusChangeSequence plus content signatures | -| AX publish lag | Prompt omits the latest character | Event and AX capture timestamps | bounded host-publication polling | -| Consuming tap ownership | Tab or another key disappears | Tap mode and accept verdict | listen-only observer plus fail-open accept tap | -| Out-of-order network stream | Old partial replaces newer state | Request IDs and stream sequence | work identity and monotonic partial policy | -| Weak caret geometry | Overlay floats or overlaps text | Geometry source/quality | quality-aware inline versus mirror | -| Insertion mismatch | Accepted text is missing or duplicated | Planned write versus fresh host state | insertion strategy plus reconciliation | -| IME composition | Acceptance re-enters composition | Input source and insertion method | IME-aware paste commit | -| Clipboard corruption | User loses clipboard contents | Pasteboard snapshot/restore logs | all-representation, change-aware restore | -| Permission drift | App works after install but not restart | TCC state and code identity | explicit permission model and dev identity | -| Backend outage | UI hangs or stale ghost remains | Timeout/cancel state | recoverable engine state and cancellation | -| Unbounded context | Latency, cost, or privacy leak | Request-size breakdown | acquisition bounds plus section budgets | -| Native/resource leak | Memory grows across sessions | model/window/task lifecycle | selected-runtime lifecycle and bounded shutdown | -| App-specific AX behavior | One host breaks while others pass | compatibility matrix | isolated fallbacks behind focus/geometry services | -| Poor observability | Team cannot reproduce reports | missing correlation/stage metadata | request-correlated structured logs | - -## A Provisional Target Architecture - -The exact types should follow the prototype language and framework, but the responsibilities should -look like this: - -~~~text -ApplicationEnvironment - owns app-lifetime services and configuration - -FocusProvider - reduces host APIs into bounded FocusSnapshot values - -InputMonitor - observes physical intent and conditionally consumes owned actions - -SuggestionCoordinator - owns state transitions, not low-level OS or model mechanics - -WorkController - owns debounce, cancellation, and current work identity - -ContextBuilder - builds one immutable bounded request - -SuggestionEngine - streams backend results behind one contract - -OutputNormalizer - enforces backend-independent display and insertion policy - -InteractionSession - owns active anchor, remaining text, type-through, and acceptance state - -OverlayController - owns non-activating presentation and geometry degradation - -Inserter - selects a host/IME-safe write strategy and reports the plan/result - -Diagnostics - correlates focus, request, stream, presentation, and acceptance -~~~ - -This is not a demand for eleven classes. It is a responsibility map. Several can begin as small value -types or protocols around working prototype code. - -## Cotabby Patterns Worth Transferring - -### One composition root - -Transfer the invariant that process-wide focus monitors, event taps, panels, and sessions have one -owner. The HyperWrite implementation may use dependency injection, an application environment, or -another composition mechanism. - -Cotabby references: - -- [CotabbyAppEnvironment.swift](../../Cotabby/App/Core/CotabbyAppEnvironment.swift) -- [AppDelegate.swift](../../Cotabby/App/Core/AppDelegate.swift) - -### Normalized bounded focus state - -Do not let every subsystem call AX independently. Reduce host state once into a value carrying -capability, text window, selection, identity, geometry quality, and surface metadata. - -Cotabby references: - -- [FocusSnapshotResolver.swift](../../Cotabby/Services/Focus/FocusSnapshotResolver.swift) -- [FocusModels.swift](../../Cotabby/Models/Focus/FocusModels.swift) - -### Explicit work identity - -Cancellation must be paired with a generation/work identifier and environment signatures. Network -responses are especially likely to finish after cancellation. - -Cotabby references: - -- [SuggestionWorkController.swift](../../Cotabby/Services/Suggestion/SuggestionWorkController.swift) -- [SuggestionCoordinator+Prediction.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator+Prediction.swift) - -### A real interaction session - -Represent active suggestion state explicitly. Store its anchor, full text, consumed portion, -trailing-text expectation, and kind. Do not infer the session from whatever the overlay currently -shows. - -Cotabby references: - -- [SuggestionInteractionState.swift](../../Cotabby/Services/Suggestion/SuggestionInteractionState.swift) -- [SuggestionSessionReconciler.swift](../../Cotabby/Support/Suggestion/SuggestionSessionReconciler.swift) - -### Small pure state machines for timing-sensitive mechanisms - -When several booleans and counters protect one timing invariant, move their transitions into a -small value while leaving scheduling and side effects with the coordinator. This makes race rules -executable without pretending the value owns event taps, timers, or windows. - -Cotabby references: - -- [SuggestionStreamingState.swift](../../Cotabby/Support/Suggestion/SuggestionStreamingState.swift) -- [PostExhaustionAcceptanceState.swift](../../Cotabby/Support/Suggestion/PostExhaustionAcceptanceState.swift) -- [SuggestionStreamingStateTests.swift](../../CotabbyTests/Support/Suggestion/SuggestionStreamingStateTests.swift) -- [PostExhaustionAcceptanceStateTests.swift](../../CotabbyTests/Support/Suggestion/PostExhaustionAcceptanceStateTests.swift) - -### Fail-open input - -Keep ordinary observation non-consuming. Enable interception only while a feature owns a key, and -swallow the key only after the action succeeds. - -Cotabby reference: - -- [InputMonitor.swift](../../Cotabby/Services/Input/InputMonitor.swift) - -### Strategy-based insertion - -Treat Unicode events, menu paste, Command-V, AX mutation, and replacements as strategies with -explicit preconditions and failure behavior. Detect composing IMEs. Reconcile afterward. - -Cotabby references: - -- [SuggestionInserter.swift](../../Cotabby/Services/Suggestion/SuggestionInserter.swift) -- [InsertionStrategySelector.swift](../../Cotabby/Support/Suggestion/InsertionStrategySelector.swift) - -### Uncertainty-aware presentation - -Carry geometry quality into presentation. If HyperWrite cannot know an exact caret, use a UI that -looks intentionally approximate instead of misaligned inline text. - -Cotabby references: - -- [CompletionRenderModePolicy.swift](../../Cotabby/Support/Presentation/CompletionRenderModePolicy.swift) -- [OverlayController.swift](../../Cotabby/Services/Presentation/OverlayController.swift) - -### Bounded context at acquisition and rendering - -Bound user text before it circulates through state, then budget it again when creating a request. -Treat each optional context source as a separate privacy boundary. - -Cotabby references: - -- [FocusSnapshotResolver.swift](../../Cotabby/Services/Focus/FocusSnapshotResolver.swift) -- [SuggestionRequestFactory.swift](../../Cotabby/Support/Suggestion/SuggestionRequestFactory.swift) -- [PromptContextSanitizer.swift](../../Cotabby/Support/Context/PromptContextSanitizer.swift) - -### Correlated observability - -Create request/session/focus identifiers early and carry them across client/server if possible. -Record state transitions and suppression reasons, not only errors. - -Cotabby references: - -- [RequestID.swift](../../Cotabby/Support/Logging/RequestID.swift) -- [SuggestionDebugLogger.swift](../../Cotabby/Services/Suggestion/SuggestionDebugLogger.swift) - -## Cotabby Decisions Not to Copy Blindly - -### Polling - -Polling is Cotabby's authoritative choice because of observed AX notification inconsistency. If -HyperWrite already has a reliable notification-plus-reconciliation design or a narrower host set, -measure it before replacing it. The transferable idea is one authoritative freshness model. - -### App-specific fallbacks - -Cotabby has Chromium, Calendar, static-run, and geometry workarounds accumulated from real hosts. -HyperWrite should add compatibility branches only for supported-product evidence, behind a narrow -boundary and regression case. - -### Llama runtime structure - -If HyperWrite is hosted-only, LlamaRuntimeManager/Core and KV-cache lifecycle are irrelevant. The -transferable ideas are serialized mutable backend state, cancellation, prewarm, and cleanup. - -### Coordinator shape - -Do not copy a large coordinator file layout. Copy the distinction between orchestration, mutable -session state, work identity, pure policy, and small mechanism-specific state machines. - -### Every Cotabby feature - -Emoji, macros, visual OCR, power profiles, multiple backends, and extensive settings are not -prerequisites for a reliable HyperWrite alpha. Additional surface area multiplies the compatibility -matrix. - -### Current secure-field acquisition - -Cotabby's secure-field generation block occurs later than the ideal acquisition boundary. HyperWrite -should make the desired privacy invariant explicit before copying focus or visual-context behavior. - -## Work Sequence for an Alpha - -This is sequencing, not a calendar. - -### Phase A: Establish the baseline - -Outputs: - -- Build and run instructions that work from a clean checkout -- A written current architecture and data-flow trace -- A list of known failures with reproduction steps -- Structured stage logging with correlation IDs -- Initial latency, crash, input, and insertion measurements -- A provisional supported-application matrix - -Why first: - -Without a baseline, architectural changes cannot be distinguished from regressions and the team will -optimize the most memorable anecdote. - -### Phase B: Define the interaction contract - -Outputs: - -- FocusSnapshot or equivalent bounded value -- Explicit capability and block reasons -- Request/work identity -- Active interaction session -- Rules for invalidation, type-through, dismissal, and acceptance -- One definition of visible-versus-accept-ready state - -Why: - -This turns implicit timing into testable state transitions. - -### Phase C: Harden focus and input - -Outputs: - -- One authoritative focus freshness mechanism -- Supported-host capability resolution -- Secure/read-only exclusion -- Fail-open input observation/interception -- Synthetic-event suppression -- Permission recovery -- Stress tests for rapid typing, field switching, and modifier changes - -Why: - -No generation improvement matters if the wrong editor is targeted or a physical key disappears. - -### Phase D: Harden request and generation - -Outputs: - -- Immutable bounded request builder -- Backend timeout, cancellation, and retry policy -- Streaming sequence/monotonicity contract -- Backend-independent output normalization -- Explicit unavailable/degraded states -- Client/server request correlation where available - -Why: - -Network/model uncertainty becomes an ordinary state instead of corrupting UI state. - -### Phase E: Harden presentation and insertion - -Outputs: - -- Non-activating overlay ownership -- Geometry quality and fallback presentation -- Session/overlay equality checks -- Host and IME-aware insertion strategy -- Post-insertion verification -- Clipboard protection if paste is used -- Multiple display, Space, full-screen, RTL, and multiline checks appropriate to scope - -Why: - -This is the point where users either trust the product or feel that it interferes with their work. - -### Phase F: Productize distribution and recovery - -Outputs: - -- Clear permission onboarding and recovery -- Stable development and production identities -- Signing/notarization/update path -- Privacy disclosure matching actual context transport -- Debug artifact policy -- Crash/hang and support-diagnostics workflow -- Release rollback plan - -Why: - -A reliable debug build is not yet a reliable product. - -### Phase G: Alpha hardening - -Outputs: - -- Executed compatibility matrix -- Adversarial and soak results -- Measured alpha gates -- Known limitations and non-goals -- Triage playbook -- Prioritized post-alpha roadmap - -Why: - -Alpha quality should be a deliberate support boundary, not “it worked in the demo.” - -## Observability Contract - -Every suggestion should carry identifiers such as: - -- focus_session_id -- request_id -- interaction_session_id -- stream_sequence -- insertion_attempt_id - -Useful stages: - -~~~text -focus_captured -focus_blocked -input_observed -host_publish_wait_started -request_built -request_dispatched -first_partial -partial_presented -final_received -result_suppressed -session_started -accept_requested -insert_attempted -insert_verified -insert_mismatch -session_invalidated -request_cancelled -~~~ - -Each stage should carry only safe metadata by default: - -- Host application and surface classification -- Focus/content signature, not raw text -- Backend and model -- Latency -- Geometry source/quality -- Context character counts by source -- Suppression or failure reason -- Insertion strategy and outcome - -Full text or screenshots should require an explicit diagnostic mode with local retention and clear -consent. If the HyperWrite server already has request IDs, the client request ID should cross the -transport boundary. - -## Failure and Recovery Matrix - -| Failure | Safe behavior | Diagnostic evidence | -| --- | --- | --- | -| No focused supported field | Hide/disable suggestion; consume nothing | capability reason | -| AX state older than input | bounded refresh/poll, then ordinary guards | event and capture ages | -| Field switch during request | cancel and reject late response | request/focus identity mismatch | -| Stream sends non-monotonic partial | hold last valid partial or reset safely | stream sequence and texts in explicit debug | -| Backend timeout | clear provisional state and show recoverable status if appropriate | backend latency and timeout classification | -| Authentication failure | stop retries, prompt settings repair | endpoint identity and HTTP classification | -| Overlay geometry invalid | suppress or use deliberate fallback UI | geometry source/quality | -| Acceptance session stale | pass the original key through | accept preflight reason | -| IME active | use verified commit strategy | input source and insertion strategy | -| Paste menu unavailable | safe fallback or fail open | menu lookup/press result | -| Host write not published | bounded refresh, invalidate speculation/session | insertion attempt and content mismatch | -| Permission revoked | tear down consuming taps and surface guidance | permission transition | -| App sleeps/wakes | refresh permissions/focus/backend and discard old sessions | lifecycle generation | -| Process termination | stop new work, cancel/flush, release resources | shutdown stage durations | - -## Test Strategy - -### Pure unit tests - -Test value-based rules without a desktop: - -- Capability/availability decisions -- Request bounds and context budgets -- Work identity -- Session transitions -- Type-through -- Word/phrase segmentation -- Stream monotonicity -- Output normalization -- Geometry-mode policy -- Insertion planning -- Clipboard restore decisions -- Retry/timeout classification - -### Coordinator tests with fakes - -Inject fake focus, engine, overlay, and inserter boundaries. Test: - -- Late result rejection -- Field switch during generation -- Partial then final suppression -- Accept preflight failure passing through -- Host publication lag -- Permission and settings changes -- Backend switching - -### Controlled host fixtures - -Build small test hosts that expose known behavior: - -- Native NSTextField and NSTextView -- Secure and read-only fields -- Multiline and mid-line content -- WebKit contenteditable -- A deliberately delayed AX publisher if feasible -- An IME/manual composition procedure - -These fixtures make regressions reproducible before testing third-party applications. - -### Supported-application matrix - -For each declared host, test: - -- Empty, beginning, middle, and end-of-line caret -- Selection and replacement -- Rapid typing and deletion -- Partial and full acceptance -- Focus switch while streaming -- Undo/redo -- Multiple windows -- Multiple displays/Spaces/full screen -- Light/dark and accessibility display settings -- Relevant IMEs and RTL if in scope -- Permission revoke/regrant -- Network loss/recovery if hosted - -### Adversarial tests - -- Responses intentionally complete out of order -- Cancellation arrives during every async stage -- User changes clipboard during paste restore -- Acceptance is pressed twice rapidly -- Host exits during capture or insertion -- Selected engine/account changes during stream -- Machine sleeps during request -- AX returns malformed or non-finite geometry -- Backend sends control tokens, prompt echoes, or an empty final - -### Soak and resource tests - -- Continuous typing and focus switching -- Repeated overlay show/hide -- Repeated backend reconnect/prewarm -- Memory stability -- Event-tap recovery -- Idle CPU/wake behavior -- Clean termination under in-flight work - -## Candidate Alpha Gates - -These are starting proposals to calibrate with Josh, not promises before a baseline. - -### Non-negotiable correctness - -- Zero swallowed non-owned keys in automated classification/replay tests -- Zero stale results applied in the adversarial focus/request suite -- Zero focus-stealing overlay events -- Zero secure-field generation requests -- Zero clipboard loss in restore and overlapping-paste tests - -### Supported-host reliability - -- A measured insertion success target, such as at least 99 percent, in the declared application - matrix -- Every failure is either a known explicit degradation or produces a correlated diagnostic trail -- No silent accept where UI claims success but the host did not mutate - -### Performance - -- Define p50 and p95 time-to-first-useful-partial after measuring the current prototype -- Define p95 input-to-overlay update separately from backend latency -- No unbounded memory growth during repeated sessions -- Idle focus monitoring remains within an agreed CPU/wake budget - -### Stability and recovery - -- No crashes or hangs in the agreed soak scenario -- Permission revoke/regrant produces a recoverable state -- Network loss, authentication failure, and timeout do not leave a stale accept-ready session -- Sleep/wake and app relaunch clear incompatible state - -### Privacy - -- Context inventory matches product disclosure -- Secure-field behavior is covered by acquisition and request tests -- Credentials use Keychain or an equivalent platform secret boundary -- Full-content diagnostics are explicit, local/authorized, and retained according to policy - -## Scope Control - -A credible alpha scope is stronger than a universal claim. - -Define: - -- Supported macOS versions -- Supported processor requirements -- Named first-class applications -- Best-effort applications -- Unsupported sensitive or unusual fields -- Supported languages and IMEs -- Supported editing shapes -- One primary generation path -- One acceptance interaction -- Explicit offline/network behavior - -Potential early non-goals: - -- Every custom editor on macOS -- Perfect inline placement when caret geometry is unavailable -- Multiple generation backends -- Screenshot context -- Complex partial-word/phrase customization -- Broad application-specific settings -- Deterministic correction, emoji, or macros - -Non-goals are not admissions of failure. They protect the reliability promise. - -## Trial Deliverables to Discuss - -A strong working trial can produce: - -- A verified architecture map of the existing HyperWrite prototype -- Correlated diagnostics for the core suggestion loop -- An agreed alpha compatibility matrix -- A written reliability contract and measurable gates -- Hardened focus/input/session/insertion boundaries -- A tested primary generation path -- Permission, signing, and distribution readiness -- Known limitations and a post-alpha roadmap - -The exact selection depends on prototype maturity. Do not commit to all deliverables before seeing -the code and current build/release state. - -## How to Answer “What Would You Do First?” - -### Short answer - -> I would get the prototype running, reproduce one successful flow and the highest-impact failures, -> and add correlated stage logging if it is missing. Then I would write down the focus, work, -> session, input-consumption, and insertion invariants. That tells us whether we need targeted -> hardening or a deeper boundary change. I would prioritize wrong-target, swallowed-key, and -> insertion failures before model quality or feature breadth. - -### Deeper answer - -> My first deliverable would be a measurable baseline and risk map, not a rewrite. I would trace one -> suggestion from the focused editor through input, request, stream, overlay, acceptance, and host -> verification. I would make each stage share a request identity and classify current failures. Then -> I would harden the smallest trusted loop for an agreed app matrix: authoritative focus snapshot, -> fail-open input, cancellation plus work identity, one active interaction session, one generation -> contract, non-activating presentation, and verified insertion. Once that loop is reliable, we can -> broaden compatibility and context without multiplying unknowns. - -## How to Answer “Why Are You a Fit?” - -> Cotabby forced me to solve the same class of problems that turn a Mac autocomplete demo into a -> product: Accessibility inconsistency, browser and Electron behavior, global input without stealing -> focus, stale async generation, streaming session state, caret geometry, IMEs, safe insertion, -> permissions, native runtime lifecycle, and correlated debugging. I would not assume HyperWrite -> needs Cotabby's exact implementation, but I know which invariants to look for, how to isolate the -> risky boundaries, and how to scope reliability around evidence from real host applications. - -## Warning Signs During Scoping - -Ask for clarification if: - -- “Works everywhere” has no application matrix -- The acceptance key is consumed before insertion success is known -- Responses have no request/focus identity -- Raw editor or screenshot context is unbounded -- The prototype has no way to correlate a user report to one suggestion -- A network retry can outlive the editor session -- Overlay visibility is treated as the source of session truth -- Synthetic input is not distinguishable from physical input -- Hosted context transport is described as fully local -- Signing/TCC/update work is deferred until after the alpha - -Do not use warning signs to criticize the prototype. Use them to ask precise questions and propose a -reliability boundary. - -## Cotabby Study References for the Session - -Review these immediately before the discussion: - -1. [Root architecture map](../../ARCHITECTURE.md) -2. [Suggestion Pipeline](../architecture/suggestion-pipeline.md) -3. [Focus and Accessibility](../architecture/focus-and-accessibility.md) -4. [Input and Insertion](../architecture/input-and-insertion.md) -5. [Inference and Prompting](../architecture/inference-and-prompting.md) -6. [Context, Privacy, and Permissions](../architecture/context-privacy-and-permissions.md) -7. [Technical Decision Question Bank](technical-question-bank.md) - -The most relevant concrete source files are: - -- [CotabbyAppEnvironment.swift](../../Cotabby/App/Core/CotabbyAppEnvironment.swift) -- [FocusTracker.swift](../../Cotabby/Services/Focus/FocusTracker.swift) -- [FocusSnapshotResolver.swift](../../Cotabby/Services/Focus/FocusSnapshotResolver.swift) -- [InputMonitor.swift](../../Cotabby/Services/Input/InputMonitor.swift) -- [SuggestionWorkController.swift](../../Cotabby/Services/Suggestion/SuggestionWorkController.swift) -- [SuggestionInteractionState.swift](../../Cotabby/Services/Suggestion/SuggestionInteractionState.swift) -- [SuggestionCoordinator+Prediction.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator+Prediction.swift) -- [SuggestionCoordinator+Acceptance.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator+Acceptance.swift) -- [SuggestionInserter.swift](../../Cotabby/Services/Suggestion/SuggestionInserter.swift) -- [OverlayController.swift](../../Cotabby/Services/Presentation/OverlayController.swift) -- [SuggestionEngineRouter.swift](../../Cotabby/Services/Runtime/SuggestionEngineRouter.swift) -- [SuggestionRequestFactory.swift](../../Cotabby/Support/Suggestion/SuggestionRequestFactory.swift) -- [SuggestionSessionReconciler.swift](../../Cotabby/Support/Suggestion/SuggestionSessionReconciler.swift) - -## Final Position - -The interview is not about proving that Cotabby is perfect. It is about proving that you can: - -- Explain a complex existing system accurately -- Identify reliability invariants from first principles -- Name tradeoffs and current debt honestly -- Investigate a prototype before prescribing a rewrite -- Convert failure modes into boundaries, tests, and measurable alpha gates -- Scope product breadth around what users can trust - -That is the bridge from Cotabby architecture to HyperWrite product engineering. diff --git a/.internal/interview-prep/technical-question-bank.md b/.internal/interview-prep/technical-question-bank.md deleted file mode 100644 index 9d0f9180..00000000 --- a/.internal/interview-prep/technical-question-bank.md +++ /dev/null @@ -1,877 +0,0 @@ -# Cotabby Technical Decision Question Bank - -## How to Use This Bank - -These are not scripts to recite word for word. Learn the reasoning, then answer in your own voice. -Every strong answer should contain: - -1. The product or platform constraint -2. The chosen design -3. The failure it prevents -4. The cost or compromise -5. A concrete file or type -6. What you would improve with more time - -When Josh asks a short question, begin with the direct decision and stop after the tradeoff. Let him -pull you deeper. When he asks for a deep dive, use the source trail to walk from event to owner to -invariant. - -## Product and System Architecture - -### 1. What is the architecture of Cotabby? - -**Strong answer** - -Cotabby is a long-lived macOS menu bar agent organized around a cross-application autocomplete state -machine. It continuously reduces Accessibility state into a bounded FocusSnapshot, observes global -input without taking focus, builds an immutable request, routes it through one of three generation -backends, normalizes the output, materializes an active suggestion session, renders through a -non-activating AppKit panel, and validates insertion back into the host. - -The model is only one stage. Most reliability work is around eventual AX state, stale async work, -input ownership, geometry quality, and proving that an insertion actually reached another process. - -The dependency graph is constructed once by CotabbyAppEnvironment. AppDelegate controls startup and -shutdown. SuggestionCoordinator owns orchestration while pure rules live in Support and side effects -live in Services. - -**Source trail** - -- [CotabbyAppEnvironment.swift](../../Cotabby/App/Core/CotabbyAppEnvironment.swift) -- [AppDelegate.swift](../../Cotabby/App/Core/AppDelegate.swift) -- [SuggestionCoordinator.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator.swift) -- [Root architecture map](../../ARCHITECTURE.md) - -**Avoid** - -Do not answer, “It watches typing and calls llama.cpp.” That omits Apple, endpoints, insertion, and -the hard cross-process parts. - -### 2. What was the hardest technical part? - -**Strong answer** - -The hardest part is maintaining one coherent editing session across independent, eventually -consistent systems. CGEvents report the physical input before many applications publish their new -text through AX. AX elements can be replaced or recycled. Model work finishes asynchronously. The -overlay is in Cotabby's process while the caret is in another process. Synthetic insertion is a -request, not proof that the host mutated. - -The response was to make freshness explicit: work IDs, focusChangeSequence, bounded content -signatures, session anchors, overlay/session equality, post-insertion sentinels, and repeated -validation after awaits. - -**Source trail** - -- [SuggestionCoordinator+Input.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator+Input.swift) -- [SuggestionWorkController.swift](../../Cotabby/Services/Suggestion/SuggestionWorkController.swift) -- [SuggestionInteractionState.swift](../../Cotabby/Services/Suggestion/SuggestionInteractionState.swift) -- [SuggestionSessionReconciler.swift](../../Cotabby/Support/Suggestion/SuggestionSessionReconciler.swift) - -### 3. Why split App, Services, Models, Support, and UI? - -**Strong answer** - -The split follows change risk and side-effect ownership. Support holds deterministic policy, Models -hold shared values and contracts, Services own OS/native/network side effects, App coordinates them, -and UI renders state. That lets us test the acceptance rule or prompt budget without installing an -event tap or creating a panel. - -The tradeoff is more types and navigation. The benefit is that platform quirks do not become -unreviewable branches inside one coordinator. The split is useful only when each boundary owns a real -invariant; I would not create a file for every small function. - -**Source trail** - -- [SuggestionAvailabilityEvaluator.swift](../../Cotabby/Support/Suggestion/SuggestionAvailabilityEvaluator.swift) -- [SuggestionSubsystemContracts.swift](../../Cotabby/Models/Suggestion/SuggestionSubsystemContracts.swift) -- [SuggestionInserter.swift](../../Cotabby/Services/Suggestion/SuggestionInserter.swift) -- [SuggestionCoordinator.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator.swift) - -### 4. Is SuggestionCoordinator too large? - -**Strong answer** - -It is still a high-complexity owner because the product loop has many transitions, but it is no -longer intended as one monolithic algorithm. Its extensions separate lifecycle, input, prediction, -and acceptance. Mutable sub-state and pure decisions have moved into SuggestionWorkController, -SuggestionInteractionState, SuggestionAvailabilityEvaluator, SuggestionRequestFactory, and -SuggestionSessionReconciler. SuggestionStreamingState now owns partial coalescing/monotonic-render -bookkeeping, while PostExhaustionAcceptanceState owns the bounded rapid-Tab transition rules. - -I would not split the coordinator into independent actors merely to reduce file size because the -transitions need one MainActor ordering domain. I would continue extracting cohesive policies and -small state machines where they can be tested without duplicating ownership. - -**Source trail** - -- [SuggestionCoordinator.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator.swift) -- [SuggestionCoordinator+Lifecycle.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator+Lifecycle.swift) -- [SuggestionCoordinator+Input.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator+Input.swift) -- [SuggestionCoordinator+Prediction.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator+Prediction.swift) -- [SuggestionCoordinator+Acceptance.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator+Acceptance.swift) -- [SuggestionStreamingState.swift](../../Cotabby/Support/Suggestion/SuggestionStreamingState.swift) -- [PostExhaustionAcceptanceState.swift](../../Cotabby/Support/Suggestion/PostExhaustionAcceptanceState.swift) - -### 4A. Why introduce domain settings without migrating persistence? - -**Strong answer** - -The settings model accumulated fields from several product areas, but SwiftUI bindings, tests, and -UserDefaults keys already formed a compatibility surface. Replacing all of that at once would mix an -ownership improvement with a risky persistence migration. - -SuggestionSettingsData groups values into general, engine, completion, context, correction, -presentation, inline-feature, and shortcut domains. SuggestionSettingsModel projects its existing -published properties into that value, snapshots derive behavior from it, and SuggestionSettingsStore -continues mapping to the established flat keys. That improves the mental model without creating two -mutable sources or invalidating saved preferences. - -The tradeoff is a temporary forwarding layer. I would migrate consumers domain by domain only where -the cohesive value improves ownership, then remove compatibility accessors when call-site evidence -says it is safe. - -**Source trail** - -- [SuggestionSettingsData.swift](../../Cotabby/Models/Settings/SuggestionSettingsData.swift) -- [SuggestionSettingsModel.swift](../../Cotabby/Models/Settings/SuggestionSettingsModel.swift) -- [SuggestionSettingsStore.swift](../../Cotabby/Support/Settings/SuggestionSettingsStore.swift) -- [SuggestionSettingsDomainTests.swift](../../CotabbyTests/Models/Settings/SuggestionSettingsDomainTests.swift) - -## Ownership and Lifecycle - -### 5. Why is there one long-lived dependency graph? - -**Strong answer** - -Cotabby owns process-wide resources: Accessibility polling, event taps, runtime memory, settings, -panels, downloads, and permission state. If a SwiftUI redraw created a second FocusTracker or -InputMonitor, we could poll twice, consume a key twice, race model reloads, or display state from a -different settings instance. - -CotabbyAppEnvironment constructs those objects once and passes narrow collaborators into -coordinators. AppDelegate starts and stops side effects at process lifecycle boundaries. The cost is -a large composition root, but that cost is visible and deterministic. - -**Source trail** - -- [CotabbyApp.swift](../../Cotabby/App/Core/CotabbyApp.swift) -- [CotabbyAppEnvironment.swift](../../Cotabby/App/Core/CotabbyAppEnvironment.swift) -- [AppDelegate.swift](../../Cotabby/App/Core/AppDelegate.swift) - -### 6. Why do AppDelegate and CotabbyAppEnvironment both retain subscriptions? - -**Strong answer** - -They own different relationships. The environment retains relationships among the objects it -constructed, such as settings changing focus cadence, power profiles selecting engines, or endpoint -identity invalidating connection state. AppDelegate retains reactions tied to process lifecycle, -such as permissions refreshing input monitoring, engine changes loading or releasing llama, and -focus changes moving activation/debug overlays. - -The distinction is ownership, not “all subscriptions belong in one file.” If the environment did not -retain its cancellables for the process lifetime, graph-internal behavior would silently stop. - -**Source trail** - -- [CotabbyAppEnvironment.swift](../../Cotabby/App/Core/CotabbyAppEnvironment.swift) -- [AppDelegate.swift](../../Cotabby/App/Core/AppDelegate.swift) -- [Lifecycle and Composition](../architecture/lifecycle-and-composition.md) - -### 7. Why is native runtime shutdown synchronous at termination? - -**Strong answer** - -The llama context and Metal/native resources outlive ordinary Swift objects and can collide with C++ -static teardown if the process exits while work is active. AppDelegate first stops new coordination -and global input, then asks LlamaRuntimeManager for a bounded synchronous shutdown. LlamaRuntimeCore -prevents new work, aborts or waits for active operations under its lifecycle condition, and releases -native state. - -The tradeoff is that termination can wait briefly. The wait must be bounded because permission flows -sometimes require a prompt quit and relaunch. - -**Source trail** - -- [AppDelegate.swift](../../Cotabby/App/Core/AppDelegate.swift) -- [LlamaRuntimeManager.swift](../../Cotabby/Services/Runtime/LlamaRuntimeManager.swift) -- [LlamaRuntimeCore.swift](../../Cotabby/Services/Runtime/LlamaRuntimeCore.swift) - -## Accessibility and Editor Compatibility - -### 8. Why poll Accessibility instead of using AXObserver notifications? - -**Strong answer** - -AX notifications are incomplete and inconsistent across AppKit, browsers, Electron, and custom -editors. Mixing notification and polling streams also creates an ordering problem: which observation -is authoritative when they disagree? - -Cotabby uses one rule: a full capture is truth for that moment, and later captures repair stale -state. Activity can request refreshNow, but that still performs a capture rather than trusting the -event payload. Adaptive backoff reduces the idle cost. - -The tradeoff is synchronous AX IPC and periodic wakeups, so deep walks are bounded, throttled, cached, -and skipped when Cotabby is disabled. - -**Source trail** - -- [FocusTracker.swift](../../Cotabby/Services/Focus/FocusTracker.swift) -- [FocusPollBackoff.swift](../../Cotabby/Support/Focus/FocusPollBackoff.swift) -- [Focus and Accessibility](../architecture/focus-and-accessibility.md) - -### 9. How do you support Chrome and Electron? - -**Strong answer** - -Browser editors often expose the focused text node only after web accessibility is primed, and -out-of-process iframes can make the system focused-element query point at the wrong process or miss -the editor. Cotabby primes Chromium accessibility, resolves the actual owning application, and has a -cursor hit-test fallback that is cached only while it remains focused and belongs to the same browser -context. - -For geometry, browsers may expose text-marker bounds rather than reliable NSRange bounds. The -resolver tries marker geometry and static text runs before field estimation. Every fallback is -revalidated so it cannot mask a real focus change. - -**Source trail** - -- [FocusTracker.swift](../../Cotabby/Services/Focus/FocusTracker.swift) -- [ChromiumAccessibilityEnabler.swift](../../Cotabby/Services/Focus/ChromiumAccessibilityEnabler.swift) -- [FocusSnapshotResolver.swift](../../Cotabby/Services/Focus/FocusSnapshotResolver.swift) -- [AXTextGeometryResolver.swift](../../Cotabby/Services/Focus/AXTextGeometryResolver.swift) - -### 10. Why classify caret geometry quality? - -**Strong answer** - -An AX rectangle is not automatically an exact caret. Direct range bounds and derived nearby-character -bounds are precise enough for inline glyphs. A field-frame estimate may only identify the text line, -and hidden TextKit layout is still an estimate of a foreign editor. - -Cotabby carries exact, derived, estimated, or layoutEstimated quality into presentation. -CompletionRenderModePolicy uses inline for exact/derived and a mirror card for estimates or a -mid-line caret. That makes uncertainty visible in the UI instead of painting text over host content. - -The tradeoff is two presentation modes and more layout policy. It is preferable to confidently wrong -inline placement. - -**Source trail** - -- [FocusModels.swift](../../Cotabby/Models/Focus/FocusModels.swift) -- [AXTextGeometryResolver.swift](../../Cotabby/Services/Focus/AXTextGeometryResolver.swift) -- [CompletionRenderModePolicy.swift](../../Cotabby/Support/Presentation/CompletionRenderModePolicy.swift) -- [OverlayController.swift](../../Cotabby/Services/Presentation/OverlayController.swift) - -### 11. Why keep AX work on MainActor if it can be slow? - -**Strong answer** - -AX element access, AppKit state, focus caches, and publication are tightly coupled to main-thread -state. MainActor gives capture and cache mutation one ordering domain and avoids unsafe concurrent use -of Core Foundation/AX objects. - -The design does not pretend this is free. It bounds text, gates parameterized calls, caches -focus-session invariants, throttles deep descendants/static runs, and uses freshness checks to avoid -duplicate captures. OCR and model generation leave MainActor because they do not need live AX -objects. - -If profiling identified a specific safe extraction that could move off actor, I would introduce a -value-type boundary first rather than pass AXUIElement into arbitrary tasks. - -**Source trail** - -- [FocusTracker.swift](../../Cotabby/Services/Focus/FocusTracker.swift) -- [FocusSnapshotResolver.swift](../../Cotabby/Services/Focus/FocusSnapshotResolver.swift) -- [AXHelper.swift](../../Cotabby/Support/Accessibility/AXHelper.swift) - -## Async State and Reliability - -### 12. Why are cancellation and work IDs both necessary? - -**Strong answer** - -Task cancellation expresses intent, but an API or native decode can finish after cancellation was -requested. If result application only checked Task.isCancelled, a previous field's completion could -still appear. - -SuggestionWorkController increments a work ID whenever debounce/generation is replaced. A result can -apply only if its captured ID remains current. The coordinator also checks focus, content, settings, -and session signatures because a current task can still be invalidated by environment changes. - -The small cost is carrying identity through async boundaries. It converts a timing assumption into an -explicit invariant. - -**Source trail** - -- [SuggestionWorkController.swift](../../Cotabby/Services/Suggestion/SuggestionWorkController.swift) -- [SuggestionCoordinator+Prediction.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator+Prediction.swift) - -### 13. How do you handle a key event arriving before AX publishes the new value? - -**Strong answer** - -The listen-only CGEvent observes the key before many hosts update AXValue. Cotabby schedules after a -short debounce, requests a fresh capture, and performs bounded host-publication polling when it knows -the host is catching up. It also tracks capture freshness to avoid paying for redundant AX walks. - -There is a ceiling. Cotabby cannot wait forever for a broken host, so it eventually proceeds through -ordinary request and stale-result guards. This is one reason the pipeline is designed around eventual -consistency rather than assuming event order. - -**Source trail** - -- [SuggestionCoordinator+Input.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator+Input.swift) -- [FocusTracker.swift](../../Cotabby/Services/Focus/FocusTracker.swift) - -### 14. Why do you revalidate at several pipeline stages? - -**Strong answer** - -Eligibility is not a lease. Passing it before debounce does not authorize work after the user changes -fields, selects text, disables an app, switches engines, or accepts part of another session. - -Cotabby gates before scheduling to avoid waste, again before request construction, and again before -partial/final application and insertion. Each boundary checks the facts that could have changed while -awaiting. - -The tradeoff is repeated-looking guard code. Consolidating all guards into one early check would be -shorter but incorrect. - -**Source trail** - -- [SuggestionAvailabilityEvaluator.swift](../../Cotabby/Support/Suggestion/SuggestionAvailabilityEvaluator.swift) -- [SuggestionCoordinator+Prediction.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator+Prediction.swift) -- [SuggestionCoordinator+Acceptance.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator+Acceptance.swift) - -### 15. Why have a separate active suggestion session? - -**Strong answer** - -A completion is not just a string. It has an anchor focus/content signature, original prefix, -trailing text, consumed character count, kind, and expected host state. The user can type through it, -accept one chunk, switch fields, undo, select text, or race AX publication. - -SuggestionInteractionState owns mutable session facts. SuggestionSessionReconciler contains pure -transition rules. This makes the lifecycle explicit and testable rather than comparing the current -overlay string opportunistically. - -**Source trail** - -- [ActiveSuggestionSession.swift](../../Cotabby/Models/Suggestion/ActiveSuggestionSession.swift) -- [SuggestionInteractionState.swift](../../Cotabby/Services/Suggestion/SuggestionInteractionState.swift) -- [SuggestionSessionReconciler.swift](../../Cotabby/Support/Suggestion/SuggestionSessionReconciler.swift) - -### 16. Why can a streamed partial be accepted before generation finishes? - -**Strong answer** - -Autocomplete is latency-sensitive. If a cumulative partial is normalized, passes seam checks, and -materializes into the same session model as a final result, the user can act on useful text while -decode continues. - -SuggestionStreamingState makes pending partials latest-wins, permits one scheduled drain at a time, -and applies the monotonic rendering rule. The coordinator owns the runloop scheduling, freshness -checks, session creation, and overlay effects. The final is still authoritative: it can replace or -suppress a partial. Acceptance or new typing cancels or supersedes remaining work through the normal -work-ID rules. - -The tradeoff is more complex session coordination, but it reduces perceived latency without creating -a second acceptance system. - -**Source trail** - -- [StreamedGhostTextPolicy.swift](../../Cotabby/Support/Suggestion/StreamedGhostTextPolicy.swift) -- [SuggestionStreamingState.swift](../../Cotabby/Support/Suggestion/SuggestionStreamingState.swift) -- [SuggestionCoordinator+Prediction.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator+Prediction.swift) -- [SuggestionInteractionState.swift](../../Cotabby/Services/Suggestion/SuggestionInteractionState.swift) - -## Input and Insertion - -### 17. How do you guarantee Cotabby does not steal user keys? - -**Strong answer** - -The steady event tap is listen-only. The only suggestion tap capable of returning nil is installed -while interception is needed, and it consumes a configured key only after the current coordinator -successfully accepts. If the overlay disappeared, the session is stale, permission changed, or -insertion validation fails, the event passes through. - -That is fail-open behavior: uncertainty favors the host application. The global toggle has a separate -tap because its ownership is independent of suggestion visibility. - -There is one tightly bounded exception to overlay visibility: after the final visible chunk is -accepted, Cotabby can briefly retain Tab ownership while the next continuation regenerates. -PostExhaustionAcceptanceState collapses rapid presses to one queued accept and uses a generation-keyed -timeout so a stale callback cannot release a newer window. Teardown or timeout always returns Tab to -the host. - -**Source trail** - -- [InputMonitor.swift](../../Cotabby/Services/Input/InputMonitor.swift) -- [SuggestionCoordinator+Acceptance.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator+Acceptance.swift) -- [PostExhaustionAcceptanceState.swift](../../Cotabby/Support/Suggestion/PostExhaustionAcceptanceState.swift) - -### 18. How do you insert text into arbitrary applications? - -**Strong answer** - -For the common short single-line case, Cotabby posts a Unicode CGEvent pair. It is app-agnostic and -does not touch the clipboard. A composing IME uses paste because a synthetic Unicode event can be -absorbed into composition. An optional strategy also pastes long or multiline chunks. - -The paste path snapshots all pasteboard representations, places the completion temporarily, tries the -target app's AX Edit > Paste menu item, falls back to Command-V, and restores the clipboard only if no -newer clipboard activity occurred. - -There is no perfect universal insertion API. The architecture makes strategy and failure explicit, -then reconciles the host's later AX state. - -**Source trail** - -- [SuggestionInserter.swift](../../Cotabby/Services/Suggestion/SuggestionInserter.swift) -- [InsertionStrategySelector.swift](../../Cotabby/Support/Suggestion/InsertionStrategySelector.swift) -- [KeyboardInputSourceMonitor.swift](../../Cotabby/Services/Input/KeyboardInputSourceMonitor.swift) - -### 19. Why does Chrome need an AX Paste menu action? - -**Strong answer** - -A synthetic Command-V posted from inside a global tap callback can be ignored in Chrome while the -physical acceptance key is still down. Pressing the application's real Paste menu item drives the -same command through its own command routing and is more reliable. Cotabby caches the menu item per -process but validates every AXPress result; failure falls back to Command-V. - -This is a targeted compatibility workaround behind SuggestionInserter rather than a browser branch in -the coordinator. - -**Source trail** - -- [SuggestionInserter.swift](../../Cotabby/Services/Suggestion/SuggestionInserter.swift) -- [AXHelper.swift](../../Cotabby/Support/Accessibility/AXHelper.swift) - -### 20. How do you prevent synthetic insertion from triggering Cotabby again? - -**Strong answer** - -SuggestionInserter tags every synthetic event with a Cotabby-specific source marker and registers a -bounded expected event burst in InputSuppressionController. InputMonitor checks the marker first and -uses the count as a compatibility fallback when event transformation loses properties. - -The count accumulates across rapid acceptances because event delivery is asynchronous. It also -expires, so it cannot become a broad period during which real user input is ignored. - -**Source trail** - -- [InputSuppressionController.swift](../../Cotabby/Services/Input/InputSuppressionController.swift) -- [InputMonitor.swift](../../Cotabby/Services/Input/InputMonitor.swift) -- [SuggestionInserter.swift](../../Cotabby/Services/Suggestion/SuggestionInserter.swift) - -## Inference and Prompting - -### 21. Why support three generation backends behind one router? - -**Strong answer** - -The product needs one suggestion contract while devices and user privacy/performance choices differ. -Apple Intelligence is integrated with the OS, the in-process llama path works with downloaded base -GGUFs, and the OpenAI-compatible path supports loopback Ollama, LAN, or public HTTPS servers. - -SuggestionEngineRouter centralizes selection, metrics, prewarm forwarding, reset, and the narrow Apple -unsupported-language fallback. Backend implementations retain their own prompt, stream, lifecycle, -and transport mechanics. - -The tradeoff is a larger test matrix. The benefit is that coordinator and session logic do not fork -per backend. - -**Source trail** - -- [SuggestionEngineRouter.swift](../../Cotabby/Services/Runtime/SuggestionEngineRouter.swift) -- [SuggestionSubsystemContracts.swift](../../Cotabby/Models/Suggestion/SuggestionSubsystemContracts.swift) - -### 22. Why does Apple use a different prompt renderer from llama? - -**Strong answer** - -The llama models are base completion models. They condition on a text sequence; wrapping them in an -instruction conversation wastes tokens and can produce scaffolding instead of a continuation. -BaseCompletionPromptRenderer budgets optional context and places the caret prefix last. - -Apple's Foundation Models API provides a first-class instructions channel. FoundationModelPromptRenderer -uses that channel for policy and keeps content in the prompt. The domain request is shared, but the -backend-appropriate representation differs. - -**Source trail** - -- [BaseCompletionPromptRenderer.swift](../../Cotabby/Support/Prompting/BaseCompletionPromptRenderer.swift) -- [FoundationModelPromptRenderer.swift](../../Cotabby/Support/Prompting/FoundationModelPromptRenderer.swift) -- [SuggestionRequestFactory.swift](../../Cotabby/Support/Suggestion/SuggestionRequestFactory.swift) - -### 23. Why are LlamaRuntimeManager and LlamaRuntimeCore separate? - -**Strong answer** - -LlamaRuntimeManager is MainActor and ObservableObject because Settings and app lifecycle need -published model/loading/error state. LlamaRuntimeCore owns mutable native pointers, tokenization, -prompt sequence reuse, decode, abort, and shutdown. Those operations must not block UI and must obey -native serialization rules. - -The core is a nonisolated class marked unchecked Sendable with explicit locks and a lifecycle -condition. That is intentional: Sendable does not make native pointers thread-safe; the implementation -must enforce the synchronization. - -The split lets the manager express product state while the core protects native correctness. - -**Source trail** - -- [LlamaRuntimeManager.swift](../../Cotabby/Services/Runtime/LlamaRuntimeManager.swift) -- [LlamaRuntimeCore.swift](../../Cotabby/Services/Runtime/LlamaRuntimeCore.swift) - -### 24. Why not make LlamaRuntimeCore a Swift actor? - -**Strong answer** - -An actor would serialize Swift entry points, but the runtime also needs synchronous abort/shutdown -coordination, condition waiting around active native operations, and a narrow lock specifically -around prompt-cache/decode state. Native callbacks and C++ lifetime do not automatically become safe -because their wrapper is an actor. - -The explicit locks make the required critical sections and shutdown protocol visible. The tradeoff is -manual synchronization and unchecked Sendable responsibility. An actor could be a valid redesign, -but only if it preserved abort and bounded shutdown without blocking an executor or leaking pointers -across isolation. - -**Source trail** - -- [LlamaRuntimeCore.swift](../../Cotabby/Services/Runtime/LlamaRuntimeCore.swift) -- [LlamaRuntimeManager.swift](../../Cotabby/Services/Runtime/LlamaRuntimeManager.swift) - -### 25. How does prompt/KV-cache reuse remain safe? - -**Strong answer** - -The core tokenizes the new prompt, finds the longest safe reusable prefix against the prepared -sequence, reuses compatible evaluated tokens, and prefills the remainder. The autocomplete lock -serializes that cache state. Field/session changes broadcast reset, model changes rebuild native -state, and incompatible prompts fall back to a fresh sequence. - -The key rule is that cache reuse is an optimization under explicit continuity, never hidden -cross-field memory. If continuity is uncertain, correctness wins over latency. - -**Source trail** - -- [LlamaRuntimeCore.swift](../../Cotabby/Services/Runtime/LlamaRuntimeCore.swift) -- [LlamaSuggestionEngine.swift](../../Cotabby/Services/Runtime/LlamaSuggestionEngine.swift) -- [SuggestionCoordinator+Prediction.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator+Prediction.swift) - -### 26. Why normalize output outside each engine? - -**Strong answer** - -Every backend can emit control tokens, echoed prompt material, reasoning scaffolding, whitespace -seams, duplicated trailing text, or empty/unsafe output. If each engine cleans independently, users -get different acceptance semantics and fixes drift. - -SuggestionTextNormalizer provides one backend-independent contract. Engine-specific adapters produce -raw text and metadata; normalization and seam guards decide what can become ghost text. - -The tradeoff is that the shared normalizer must not accidentally encode quirks so narrowly that it -damages another backend. Detailed suppression reasons and tests help keep that visible. - -**Source trail** - -- [SuggestionTextNormalizer.swift](../../Cotabby/Support/Suggestion/SuggestionTextNormalizer.swift) -- [CompletionSeamGuard.swift](../../Cotabby/Support/Suggestion/CompletionSeamGuard.swift) -- [LlamaSuggestionEngine.swift](../../Cotabby/Services/Runtime/LlamaSuggestionEngine.swift) -- [FoundationModelSuggestionEngine.swift](../../Cotabby/Services/Runtime/FoundationModelSuggestionEngine.swift) - -## Context and Privacy - -### 27. What data does Cotabby use, and when can it leave the Mac? - -**Strong answer** - -A request can contain bounded text before and after the caret, app/window/domain/placeholder surface -metadata, user rules and extended context, a relevance-filtered clipboard excerpt, and cleaned -screenshot OCR when enabled and permitted. - -Apple Intelligence and the in-process llama engine generate on the Mac. The endpoint engine sends the -bounded constructed request to the configured server. Loopback stays on the machine, LAN leaves the -process/machine boundary to a local server, and public HTTPS leaves the local network. Credentials are -stored in Keychain, and insecure public HTTP is rejected. - -Debug mode is a separate privacy boundary because it can write full prompts, completions, AX dumps, -and screenshot/OCR artifacts locally. - -**Source trail** - -- [SuggestionRequestFactory.swift](../../Cotabby/Support/Suggestion/SuggestionRequestFactory.swift) -- [OpenAICompatibleEndpointModels.swift](../../Cotabby/Models/Runtime/OpenAICompatibleEndpointModels.swift) -- [CotabbyDebugOptions.swift](../../Cotabby/Support/Logging/CotabbyDebugOptions.swift) -- [Context, Privacy, and Permissions](../architecture/context-privacy-and-permissions.md) - -### 28. How does visual context work, and why not summarize OCR with another model? - -**Strong answer** - -VisualContextCoordinator owns one field-scoped session. WindowScreenshotService captures a compact -ScreenCaptureKit region, ScreenTextExtractor runs Vision OCR with per-line confidence, OCRTextHygiene -removes corruption and UI chrome, and ScreenshotContextGenerator sanitizes and caps the excerpt. - -There is no model summarization stage. A second generation adds latency, can hallucinate, complicates -privacy, and can destroy literal context a base completion model could condition on directly. The -tradeoff is that deterministic hygiene must handle OCR noise well. - -**Source trail** - -- [VisualContextCoordinator.swift](../../Cotabby/Services/Visual/VisualContextCoordinator.swift) -- [WindowScreenshotService.swift](../../Cotabby/Services/Visual/WindowScreenshotService.swift) -- [ScreenTextExtractor.swift](../../Cotabby/Services/Visual/ScreenTextExtractor.swift) -- [OCRTextHygiene.swift](../../Cotabby/Support/Context/OCRTextHygiene.swift) -- [ScreenshotContextGenerator.swift](../../Cotabby/Services/Visual/ScreenshotContextGenerator.swift) - -### 29. Are secure fields never captured? - -**Strong answer** - -No. The accurate current guarantee is narrower: secure fields are marked blocked, so they cannot -schedule generation, display a suggestion, accept text, or open emoji/macro capture. Their context -cannot reach a generation engine. - -However, FocusSnapshotResolver currently constructs a bounded FocusedInputSnapshot before returning -blocked capability, and visual-capture eligibility deliberately ignores capability, including secure -fields. With Screen Recording and visual context enabled, screenshot/OCR can run; explicit debug mode -can persist that capture. - -I would call that privacy debt rather than defend it. To reach a true no-capture guarantee, I would -move secure detection ahead of value/context construction and make visual eligibility reject secure -capability, then add tests proving no AX text, screenshot, OCR, or debug artifact is produced. - -**Source trail** - -- [FocusSnapshotResolver.swift](../../Cotabby/Services/Focus/FocusSnapshotResolver.swift) -- [SuggestionAvailabilityEvaluator.swift](../../Cotabby/Support/Suggestion/SuggestionAvailabilityEvaluator.swift) -- [VisualContextCoordinator.swift](../../Cotabby/Services/Visual/VisualContextCoordinator.swift) - -**Why this answer matters** - -An honest precise limitation demonstrates more trustworthiness than repeating an outdated privacy -claim. - -## Presentation and Product Decisions - -### 30. Why use AppKit panels instead of pure SwiftUI? - -**Strong answer** - -Cotabby needs a borderless panel above another app, on every Space and full-screen environment, -without becoming key, accepting mouse events, entering the window cycle, or stealing editor focus. -Those are NSPanel and NSWindow behaviors. - -OverlayController owns the panel and hosts typed SwiftUI content for the ghost or mirror view. This -uses SwiftUI where it is strong and AppKit for window semantics. The tradeoff is bridging two UI -systems, but a pure SwiftUI scene does not provide the required cross-application panel control. - -**Source trail** - -- [OverlayController.swift](../../Cotabby/Services/Presentation/OverlayController.swift) -- [SuggestionOverlayPresenter.swift](../../Cotabby/Services/Suggestion/SuggestionOverlayPresenter.swift) - -### 31. Why have emoji and macros inside an autocomplete app? - -**Strong answer** - -They reuse the same cross-application focus and insertion infrastructure but are deterministic, -low-latency productivity features. They do not call the model. Pure trigger state machines decide -whether colon or slash capture is active, and InlineCommandCoordinator arbitrates the one consuming -input slot. - -Architecturally, they demonstrate why InputMonitor should expose semantic capture ownership rather -than hard-code suggestion logic. The risk is feature interference, so their sigils are disjoint, -capture is pinned to one focus sequence, and the suggestion coordinator stands down when either -feature owns the event. - -**Source trail** - -- [InlineCommandCoordinator.swift](../../Cotabby/App/Coordinators/InlineCommandCoordinator.swift) -- [EmojiPickerController.swift](../../Cotabby/App/Coordinators/EmojiPickerController.swift) -- [MacroController.swift](../../Cotabby/App/Coordinators/MacroController.swift) -- [EmojiTriggerStateMachine.swift](../../Cotabby/Support/Emoji/EmojiTriggerStateMachine.swift) -- [MacroTriggerStateMachine.swift](../../Cotabby/Support/Macros/MacroTriggerStateMachine.swift) - -## Testing, Critique, and Engineering Judgment - -### 32. How do you test a system that depends on global macOS behavior? - -**Strong answer** - -I separate deterministic policy from OS boundaries. Work identity, request construction, -normalization, session reconciliation, trigger machines, geometry policy, prompt rendering, and -insertion planning have unit-testable value inputs. Coordinators depend on narrow protocols and fakes. - -Real AX, event taps, host publication, IMEs, browser behavior, signing, and TCC still require -integration/manual compatibility testing. Unit tests reduce the state-space before that matrix; they -do not pretend to replace it. - -CI independently checks compilation, tests, SwiftLint, and XcodeGen synchronization. - -**Source trail** - -- [CotabbyTests](../../CotabbyTests) -- [SuggestionSubsystemContracts.swift](../../Cotabby/Models/Suggestion/SuggestionSubsystemContracts.swift) -- [tests workflow](../../.github/workflows/tests.yml) -- [XcodeGen workflow](../../.github/workflows/xcodegen.yml) - -### 33. What are the biggest pieces of technical debt? - -**Strong answer** - -I would name debt precisely: - -1. Secure-field acquisition is broader than the privacy promise we want. -2. SuggestionCoordinator remains complex even after extracting pure rules and state owners. -3. Accessibility compatibility contains necessary app-specific branches that need a documented - compatibility matrix and regression harness. -4. Unit coverage is strong, but a repeatable end-to-end host compatibility harness is still the next - step for proving AX, input, presentation, and insertion together. -5. Endpoint support expands the privacy and availability matrix beyond the original on-device story. - -I would not propose a rewrite. I would first fix privacy-boundary tests and documentation, strengthen -integration replay/compatibility coverage, and continue extracting cohesive policy from the -coordinator only when ownership becomes clearer. - -**Source trail** - -- [SuggestionAvailabilityEvaluator.swift](../../Cotabby/Support/Suggestion/SuggestionAvailabilityEvaluator.swift) -- [SuggestionCoordinator.swift](../../Cotabby/App/Coordinators/SuggestionCoordinator.swift) -- [FocusSnapshotResolver.swift](../../Cotabby/Services/Focus/FocusSnapshotResolver.swift) -- [SuggestionStreamingState.swift](../../Cotabby/Support/Suggestion/SuggestionStreamingState.swift) -- [PostExhaustionAcceptanceState.swift](../../Cotabby/Support/Suggestion/PostExhaustionAcceptanceState.swift) - -### 34. What would you do differently if you started Cotabby again? - -**Strong answer** - -I would define the interaction session and reliability invariants earlier: focus identity, -request/work identity, overlay/session equality, fail-open input, and post-insertion verification. I -would also establish an application compatibility matrix and correlated structured logging before -adding many app-specific fallbacks. - -I would keep the same broad boundaries—one dependency graph, normalized focus model, pure rules, -backend router, native core, and AppKit panel ownership—but make privacy acquisition gates and test -seams first-class from the first version. - -The useful lesson is not that the current code is wrong; it is that the difficult state-machine -properties became visible through real host applications and should be explicit earlier in the next -product. - -### 35. How do you know Cotabby is reliable? - -**Strong answer** - -Reliability is not one metric. I would separate: - -- Input integrity: no non-owned key is swallowed -- Freshness: no result applies to the wrong focus or text -- Insertion correctness: accepted text matches the planned mutation -- Presentation correctness: visible text matches session state and is anchored safely -- Availability: permissions and selected runtime degrade explicitly -- Latency: time to first useful partial and final result -- Resource behavior: idle AX cost, memory, model switch cleanup, shutdown -- Privacy: context acquisition, persistence, and transport match disclosure - -Cotabby has explicit guards and structured request-correlated logs for these, plus unit tests around -pure invariants. The next maturity step is a formal compatibility matrix and repeatable end-to-end -host harness that reports these dimensions per application. - -**Source trail** - -- [RequestID.swift](../../Cotabby/Support/Logging/RequestID.swift) -- [SuggestionDebugLogger.swift](../../Cotabby/Services/Suggestion/SuggestionDebugLogger.swift) -- [SuggestionSessionReconciler.swift](../../Cotabby/Support/Suggestion/SuggestionSessionReconciler.swift) -- [InputMonitor.swift](../../Cotabby/Services/Input/InputMonitor.swift) - -## HyperWrite Scoping Questions - -### 36. How would you approach turning the HyperWrite Mac prototype into a reliable alpha? - -**Strong answer** - -I would begin by observing the current prototype and defining its core interaction contract before -rewriting anything. For each suggestion I need to know: - -- What focused editor state is considered authoritative? -- What makes a request current or stale? -- Which component owns the active suggestion? -- Which keys may be consumed, and under what success condition? -- How is accepted text inserted and verified? -- What are the supported applications and explicit degradation modes? -- What context leaves the machine? -- Which correlated events let us explain a failure? - -I would instrument that loop first, then harden focus/input/insertion and stale-work invariants, -because model quality cannot compensate for dropped keys or text inserted into the wrong field. I -would preserve working prototype components behind narrow interfaces and replace only boundaries -whose failure data justifies it. - -The alpha should have a declared compatibility matrix and measurable gates for input integrity, -stale-result rejection, insertion success, crash-free operation, latency, memory, and privacy—not a -claim that every macOS text field works. - -**Further preparation** - -- [HyperWrite Reliability Translation](hyperwrite-reliability-translation.md) - -### 37. Would you copy Cotabby's architecture into HyperWrite? - -**Strong answer** - -I would transfer invariants, not copy the repository. The reusable ideas are one ownership graph, -normalized focus state, explicit work identity, a real interaction session, fail-open input, -strategy-based insertion, backend-independent normalization, geometry-quality-aware presentation, -bounded context, and request-correlated observability. - -I would not automatically copy polling cadence, app-specific AX fallbacks, local llama lifecycle, -prompt format, settings complexity, or the current coordinator shape. Those depend on HyperWrite's -prototype, backend, product promise, and supported application set. - -The first technical session should determine which failure modes are actually present. - -### 38. What would you prioritize if the trial is constrained? - -**Strong answer** - -I would prioritize the smallest loop that users can trust: - -1. Reliable focused-editor snapshot in the agreed application set -2. Fail-open input handling -3. Stale-request cancellation and identity -4. One validated generation path -5. Overlay/session consistency -6. Verified insertion, including IME policy -7. Correlated observability and repeatable compatibility tests -8. Permission/onboarding and distributable signing - -I would defer broad app claims, multiple speculative backends, elaborate settings, and optional -context until the core loop has measured reliability. That sequencing creates an alpha we can learn -from rather than a wide demo with unexplained failures. - -## Rapid-Fire Follow-Ups - -Use these to test whether you understand the answers rather than memorizing them: - -- What exact state invalidates one suggestion? -- What is the narrowest possible stale-result check, and why is it insufficient alone? -- Which code can legally return nil from a CGEvent tap? -- What happens when the visible overlay and session tail disagree? -- Which cache is allowed to survive ordinary typing? -- Which cache must reset on a field switch? -- What is the difference between local process, local machine, LAN, and public endpoint privacy? -- Which synchronous operation is most likely to hurt typing latency? -- Which native object is least safe to move across isolation? -- What proves that an insertion succeeded? -- Which bug would request_id let you diagnose that a plain error string would not? -- Which current product statement would you correct before a public endpoint launch? - -If you cannot answer a follow-up with a file and an invariant, return to the corresponding mastery -area in [the study path](README.md). From 766a878be76c70ea46df5268914a4dd0ec656e57 Mon Sep 17 00:00:00 2001 From: Jacob Fu <141651335+FuJacob@users.noreply.github.com> Date: Fri, 17 Jul 2026 22:40:28 -0700 Subject: [PATCH 3/3] Clarify internal documentation scope --- .claude/CLAUDE.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index 6f25409e..43f98409 100644 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -5,4 +5,4 @@ `AGENTS.md` is the canonical coding-agent instruction file. Keep shared workflow, safety, debugging, validation, and architecture rules there so Claude Code and other agents receive the same guidance without duplicated copies drifting apart. `ARCHITECTURE.md` is the ten-minute system map, while -`.internal/` contains the committed deep-dive and interview-preparation documentation. +`.internal/` contains local, gitignored deep-dive and interview-preparation documentation.