Simplify inference to one sequence

.github/copilot-instructions.md

@@ -1,30 +1,41 @@
-<!-- ROCKETRIDE:BEGIN -->
+# CotabbyInference Coding Instructions
 
-# RocketRide — AI Pipeline Builder
+CotabbyInference is Cotabby's product-specific C++ wrapper around llama.cpp. Preserve its narrow
+single-sequence boundary unless a shipping Cotabby requirement explicitly changes it.
 
-Use RocketRide when building AI pipelines, document processing, RAG systems, or data integration.
+## Ownership
 
-## Documentation
+- `CotabbyInferenceEngine` owns one model, one context, and at most one active sequence.
+- Cotabby's Swift `LlamaRuntimeCore` owns generation serialization, token budgets, prompt/cache
+  compatibility, streaming, and lifecycle admission.
+- The native engine owns sampler state, seed/pending-token handoff, KV mutation, token masks,
+  cancellation observation, and native resource release.
+- Do not add editor, Accessibility, UI, or hosted-service concepts to this package.
 
-Full docs: `.rocketride/docs/`
+## Native Safety
 
-**Read the relevant doc(s) before generating any RocketRide code.**
+- Keep the public header free of llama.cpp and ggml types.
+- Treat `SampleResult.piece` as borrowed memory that expires when sequence storage changes.
+- Serialize every llama context mutation through `decode_mutex`.
+- Keep cancellation atomic and nonblocking; an active `llama_decode` is not preemptible.
+- Never destroy a sequence or unload the model while another non-cancellation operation is using it.
+- Preserve the changing external sequence ID even though the internal llama sequence slot is fixed
+  at zero; it protects replacement sequences from late cancellation.
 
-| File                              | Read when...                                                      |
-| --------------------------------- | ----------------------------------------------------------------- |
-| ROCKETRIDE_README.md              | Starting any RocketRide work — overview + mandatory setup steps   |
-| ROCKETRIDE_QUICKSTART.md          | Writing first pipeline — complete working examples (Python & TS)  |
-| ROCKETRIDE_PIPELINE_RULES.md      | Defining pipelines — structure, lane wiring, config rules         |
-| ROCKETRIDE_COMPONENT_REFERENCE.md | Choosing/configuring components — all providers and config fields |
-| ROCKETRIDE_COMMON_MISTAKES.md     | Before finalizing — known pitfalls to avoid                       |
-| ROCKETRIDE_python_API.md          | Python SDK — client methods, types, patterns                      |
-| ROCKETRIDE_typescript_API.md      | TypeScript SDK — client methods, types, patterns                  |
-| ROCKETRIDE_OBSERVABILITY.md       | Consuming runtime logs, lifecycle events, and pipeline traces     |
+## Change Strategy
 
-## Before Writing ANY RocketRide Code
+- Add only APIs used by the current Cotabby integration or backed by a concrete consumer plan.
+- Prefer removing obsolete experimental surfaces over retaining speculative generality.
+- Treat the llama.cpp binary URL/checksum as a native compatibility boundary.
+- Update `README.md`, package tests, and local `.internal/` notes when architecture changes.
+- Keep `.internal/` gitignored; it is local study material, not PR content.
 
-1. Read `.rocketride/docs/ROCKETRIDE_README.md` for mandatory setup requirements
-2. Read the relevant API doc (Python or TypeScript) for your language
-3. Read `.rocketride/docs/ROCKETRIDE_PIPELINE_RULES.md` + `.rocketride/docs/ROCKETRIDE_COMPONENT_REFERENCE.md`
-4. Read `.rocketride/docs/ROCKETRIDE_COMMON_MISTAKES.md` before finalizing
-<!-- ROCKETRIDE:END -->
+## Validation
+
+Run `swift test` for compilation and no-model contracts. When a GGUF is available, also run:
+
+~~~bash
+COTABBY_TEST_MODEL_PATH=/absolute/path/model.gguf swift test
+~~~
+
+For public API changes, build the current Cotabby app against the local package before merging.

.gitignore

@@ -8,3 +8,4 @@ DerivedData/
 .netrc
 .rocketride/
 .env
+.internal/

Package.swift

@@ -33,18 +33,5 @@ let package = Package(
                 .interoperabilityMode(.Cxx),
             ]
         ),
-        // Phase 0 spike: standalone executable that compares aggregate decode
-        // tok/s between the current "separate llama_context per sequence"
-        // architecture and a prospective shared-context-with-batched-decode
-        // architecture. Used to decide whether the Phase 1 refactor is worth
-        // doing on M-series + Metal. Not linked into CotabbyInference itself.
-        .executableTarget(
-            name: "CotabbyInferenceBench",
-            dependencies: ["llama-cpp"],
-            path: "Sources/CotabbyInferenceBench",
-            cxxSettings: [
-                .unsafeFlags(["-std=c++17"]),
-            ]
-        ),
     ]
 )

README.md

@@ -1,81 +1,93 @@
 # CotabbyInference
 
-A C++ middleware layer for running LLM inference on-device, built on top of [llama.cpp](https://github.com/ggml-org/llama.cpp). Designed as the inference backend for [Cotabby](https://github.com/fujacob/cotabby), a macOS AI assistant.
+CotabbyInference is Cotabby's narrow C++ boundary around
+[llama.cpp](https://github.com/ggml-org/llama.cpp). It owns the mapped GGUF model, one llama
+context, one autocomplete sequence, sampler state, KV-cache mutation, cancellation, and the
+token-level signals consumed by the macOS app.
 
-[![MIT License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
+The package deliberately does not own prompts, request identity, streaming order, normalization,
+editor focus, overlays, or insertion. Those remain product responsibilities in Cotabby.
 
-## Why CotabbyInference?
+## Current Architecture
 
-llama.cpp is powerful but low-level. CotabbyInference sits on top of it and provides:
+~~~text
+Cotabby LlamaRuntimeCore
+  -> CotabbyInferenceEngine
+      -> one mapped llama_model
+      -> one llama_context / KV allocation
+      -> zero or one SequenceState using llama seq_id 0
+      -> one sampler chain
+~~~
 
-- **Concurrent sequences** -- run up to 4 independent inference streams at once (e.g. autocomplete + summarization), each with its own context, sampler, and sampling config. No shared decode mutex, no contention.
-- **Per-sequence isolation** -- every sequence owns its own `llama_context` and sampler chain. One sequence can be cancelled, trimmed, or destroyed without affecting the others.
-- **Thread-safe cancellation** -- cancel any running sequence from any thread via an atomic flag. The next decode or sample call returns immediately with a `cancelled` status.
-- **KV cache control** -- trim the KV cache per-sequence to reuse prompt prefixes without re-decoding. Useful for autocomplete where the user keeps typing.
-- **Clean Swift interop** -- the public header uses PIMPL to hide all llama.cpp internals. Swift consumers import a single `CotabbyInference` module with no transitive C++ dependencies. The engine class is move-only for `~Copyable` compatibility in Swift 6.2.
-- **Zero-config GPU** -- pass `-1` for GPU layers and the engine offloads everything it can to Metal. No manual layer counting.
+Cotabby serializes generation and prefill through its runtime lock, so the middleware does not
+reserve unused secondary sequence capacity or run a batching worker. Prompt decode, feedback decode,
+KV trim, and sequence destruction use one native context mutex. Cancellation is the intentional
+cross-thread operation and uses a one-way atomic flag.
 
-## Requirements
+The public sequence ID changes whenever a sequence is recreated even though llama's internal slot
+is always zero. That prevents a late cancellation from accidentally targeting a replacement
+sequence.
 
-- macOS 14+
-- Swift 6.2+
-- Xcode 26+
+## Responsibilities
 
-## Installation
-
-Add CotabbyInference to your `Package.swift`:
-
-```swift
-dependencies: [
-    .package(url: "https://github.com/FuJacob/cotabbyinference.git", from: "0.2.0"),
-],
-targets: [
-    .target(
-        name: "YourTarget",
-        dependencies: [
-            .product(name: "CotabbyInference", package: "cotabbyinference"),
-        ],
-        swiftSettings: [
-            .interoperabilityMode(.Cxx),
-        ]
-    ),
-]
-```
-
-## Usage
-
-```swift
-import CotabbyInference
+- Load and unload one memory-mapped GGUF model.
+- Allocate one context with exactly the configured token window.
+- Tokenize Cotabby's base-model continuation prompts.
+- Build the sampler chain and precompute invalid-token, line-break, and word-boundary masks.
+- Decode a prompt and capture its first seed token while the final logits row is live.
+- Return one sampled UTF-8 piece at a time.
+- Expose sampled EOS, raw-argmax EOG intent, cancellation, and optional log-probability.
+- Trim the sequence KV cache for verified prefix reuse.
+- Release sampler, context, model, and llama backend resources in order.
 
-var engine = CotabbyInferenceEngine()
+## Swift Package
 
-// Load a GGUF model (-1 for all GPU layers, 2048 context, 512 batch)
-let status = engine.loadModel("/path/to/model.gguf", -1, 2048, 512)
+The package contains:
 
-// Tokenize
-let prompt = "The quick brown fox"
-let tokens = engine.tokenize(prompt, Int32(prompt.utf8.count))
+- `llama-cpp`: checksum-pinned binary build `b9310`;
+- `CotabbyInferenceEngine`: the C++ library imported by Cotabby through Swift C++ interop;
+- `CotabbyInferenceTests`: no-model contract tests and optional GGUF-backed integration tests.
+
+Cotabby currently consumes the package's `main` branch through `project.yml` and records an exact
+revision in `Package.resolved`.
+
+## Basic Usage
+
+~~~swift
+import CotabbyInference
+
+var engine = CotabbyInferenceEngine()
+guard engine.loadModel("/path/to/model.gguf", -1, 2048, 512) == .ok else {
+    fatalError("Model load failed")
+}
+defer { engine.unloadModel() }
 
-// Create a sequence with sampling parameters
 let config = SamplingConfig(
-    max_prediction_tokens: 64,
-    temperature: 0.7,
-    top_k: 40,
-    top_p: 0.95,
-    min_p: 0.05,
-    repetition_penalty: 1.1,
-    seed: 0
+    temperature: 0.1,
+    top_k: 20,
+    top_p: 0.7,
+    min_p: 0.08,
+    repetition_penalty: 1.05,
+    seed: 0x00C0_FFEE,
+    single_line: true
 )
-let seqId = engine.createSequence(config)
 
-// Decode prompt into KV cache
-var tokenArray = Array(tokens)
-engine.decodePrompt(seqId, &tokenArray, Int32(tokenArray.count), 0)
+let sequenceID = engine.createSequence(config)
+guard sequenceID >= 0 else {
+    fatalError("A sequence is already active or the model is unavailable")
+}
+defer { engine.destroySequence(sequenceID) }
+
+let prompt = "The quick brown fox"
+var tokens = Array(engine.tokenize(prompt, Int32(prompt.utf8.count)))
+guard engine.decodePrompt(sequenceID, &tokens, Int32(tokens.count), 0) == .ok else {
+    fatalError("Prompt decode failed")
+}
 
-// Sample tokens
-while true {
-    let result = engine.sampleNext(seqId)
-    if result.is_eos { break }
+// The caller owns the generation budget.
+for _ in 0 ..< 8 {
+    let result = engine.sampleNext(sequenceID)
+    if result.is_eos || result.was_cancelled { break }
 
     if let piece = result.piece, result.piece_length > 0 {
         let text = String(
@@ -88,48 +100,60 @@ while true {
         print(text, terminator: "")
     }
 }
+~~~
 
-// Cleanup
-engine.destroySequence(seqId)
-engine.unloadModel()
-```
+`SampleResult.piece` is borrowed sequence storage. Copy it before another sampling call or sequence
+destruction.
 
-### Running multiple sequences concurrently
+## Generation Semantics
 
-Each sequence is fully independent -- different sampling configs, different prompts, different lifetimes:
+`decodePrompt` decodes the prompt and immediately samples one seed token. The first `sampleNext`
+returns that saved seed without another decode. Each later call feedback-decodes the previously
+returned token into KV and samples the next token.
 
-```swift
-// Autocomplete: low temperature, short output
-let autocompleteConfig = SamplingConfig(
-    max_prediction_tokens: 8, temperature: 0.1,
-    top_k: 20, top_p: 0.7, min_p: 0.08,
-    repetition_penalty: 1.05, seed: 42
-)
-let seqA = engine.createSequence(autocompleteConfig)
+Cotabby controls the maximum token count in Swift. The engine controls token selection and reports:
 
-// Summary: higher temperature, longer output
-let summaryConfig = SamplingConfig(
-    max_prediction_tokens: 256, temperature: 0.5,
-    top_k: 40, top_p: 0.95, min_p: 0.05,
-    repetition_penalty: 1.4, seed: 0
-)
-let seqB = engine.createSequence(summaryConfig)
+- `is_eos`: the sampled token is an end-of-generation token;
+- `was_cancelled`: the native sequence cancellation flag was observed;
+- `argmax_is_eog`: the raw model distribution most strongly wanted to stop even if stochastic
+  sampling selected visible text;
+- `logprob`: the selected token's raw-model log-probability when enabled.
+
+## KV Reuse and Cancellation
+
+`trimKV` removes a suffix from fixed llama sequence slot zero and invalidates any saved seed or
+pending feedback token. Cotabby independently validates request continuity, UTF-8 prefix, token
+prefix, and sampling compatibility before calling it.
+
+`cancelSequence` is thread-safe and nonblocking. Prompt decode checks cancellation between chunks;
+sample generation checks before work and after feedback decode. An active llama decode is not
+preempted mid-call. Cotabby destroys a natively cancelled sequence because the flag is intentionally
+one-way.
 
-// Both run against the same loaded model, with separate contexts.
-// Cancel one without affecting the other:
-engine.cancelSequence(seqA)
-```
+## Testing
 
-## Architecture
+Run compile-time and no-model contracts:
 
-CotabbyInference gives each sequence its own `llama_context` and sampler chain. This means:
+~~~bash
+swift test
+~~~
 
-- No shared decode mutex -- sequences never block each other
-- Clean cancellation via per-sequence atomic flags
-- Independent KV caches that can be trimmed separately
-- Up to 4 concurrent sequences (the memory overhead per context is ~2-4 MB for small models)
+Run the full native path with a local GGUF:
 
-The public C++ API uses PIMPL to keep all llama.cpp headers out of the public interface. Swift consumers link against the `CotabbyInference` module only -- no need to deal with `llama.h`, `ggml.h`, or any transitive C dependencies.
+~~~bash
+COTABBY_TEST_MODEL_PATH=/absolute/path/model.gguf swift test
+~~~
+
+The model-backed suite covers single-sequence admission/replacement, prompt decode, sampling, KV
+trim, cancellation, mid-word continuation, optional log-probability, scaffolding-token masking, and
+argmax-EOG behavior. CI does not currently provide a GGUF, so these tests skip there unless the
+environment variable is configured.
+
+## Requirements
+
+- macOS 14+
+- Swift 6.2+
+- Xcode 26+
 
 ## License