Update Cotabby for single-sequence inference#803
Conversation
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| private static func samplingConfig(from options: LlamaGenerationOptions) -> SamplingConfig { | ||
| SamplingConfig( | ||
| max_prediction_tokens: Int32(options.maxPredictionTokens), | ||
| temperature: Float(options.temperature), | ||
| top_k: Int32(options.topK), | ||
| top_p: Float(options.topP), | ||
| min_p: Float(options.minP), | ||
| repetition_penalty: Float(options.repetitionPenalty), | ||
| seed: options.seed ?? Self.defaultSamplerSeed, | ||
| single_line: options.singleLine | ||
| ) | ||
| // Assign the fields after default construction so the app remains source-compatible while | ||
| // CotabbyInference removes native configuration fields that Swift never consumed. C++ | ||
| // aggregate memberwise initializers otherwise require every imported field at the call site. | ||
| var config = SamplingConfig() | ||
| config.temperature = Float(options.temperature) | ||
| config.top_k = Int32(options.topK) | ||
| config.top_p = Float(options.topP) | ||
| config.min_p = Float(options.minP) | ||
| config.repetition_penalty = Float(options.repetitionPenalty) | ||
| config.seed = options.seed ?? Self.defaultSamplerSeed | ||
| config.single_line = options.singleLine | ||
| return config | ||
| } |
There was a problem hiding this comment.
max_prediction_tokens silently dropped from C++ boundary config. The old call explicitly passed Int32(options.maxPredictionTokens) as max_prediction_tokens; default-constructing SamplingConfig() leaves that field at its C++ aggregate zero-initialization default (0). The Swift for _ in 0 ..< options.maxPredictionTokens loop at line 373 is the authoritative budget owner and is unchanged, so this is fine as long as sampleNext is stateless with respect to max_prediction_tokens. If a future C++ revision adds an internal guard in sampleNext that reads the sequence's stored config limit and short-circuits on 0, every decode would silently produce zero tokens. Since the PR validates builds against both package revisions and the comment calls this field "never consumed," this is currently safe, but worth flagging for reviewers of the companion CotabbyInference change.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
1 participant
Summary
Make the Swift llama bridge source-compatible with the simplified one-sequence CotabbyInference API, and document that the app owns one autocomplete sequence and the output-token budget. Default-constructing the C++ aggregate keeps this app change compatible with both the old and new package revisions, allowing it to land first without breaking main.
Validation
xcodegen generate— project regenerated with no project-file diff.xcodebuild -project Cotabby.xcodeproj -scheme Cotabby -destination 'platform=macOS' build -derivedDataPath build/DerivedData -quietagainst CotabbyInferencemain— exit 0.8f3979donchore/single-sequence-runtime— exit 0.Linked issues
None.
Risk / rollout notes
This is a compatibility bridge for the coordinated CotabbyInference cleanup. Runtime generation limits remain owned by the existing Swift loop; no user-facing policy changes.
Greptile Summary
This PR adapts
LlamaRuntimeCoreto stay source-compatible with a forthcoming CotabbyInference API simplification that removes C++ aggregate fields Swift never needed, and documents the single-sequence ownership contract in both AGENTS.md and ARCHITECTURE.md.samplingConfignow default-constructsSamplingConfig()and assigns only the fields Swift actually uses, dropping the explicitmax_prediction_tokenspass-through; the Swiftfor _ in 0 ..< options.maxPredictionTokensloop at line 373 remains the sole token-budget owner.Confidence Score: 4/5
Safe to merge; the core runtime behaviour is unchanged and the compatibility bridge is validated against both package revisions.
The only behavioural delta at the C++ boundary is that max_prediction_tokens now arrives as the aggregate zero-initialized default instead of options.maxPredictionTokens. The Swift token loop remains the authoritative budget enforcer, so this is inert today. A future C++ change that reads the stored config limit inside sampleNext could silently produce zero tokens, worth keeping in mind when reviewing the companion CotabbyInference PR.
The companion CotabbyInference change (commit 8f3979d on chore/single-sequence-runtime) should be reviewed in tandem with LlamaRuntimeCore.swift to confirm that sampleNext never reads max_prediction_tokens from the sequence config.
Important Files Changed
Sequence Diagram
%%{init: {'theme': 'neutral'}}%% sequenceDiagram participant Swift as Swift (LlamaRuntimeCore) participant Engine as CotabbyInferenceEngine (C++) participant KV as llama.cpp KV Cache (slot 0) Note over Swift,KV: Single autocomplete sequence lifecycle Swift->>Swift: samplingConfig(from: options) — SamplingConfig() + field assign Swift->>Engine: createSequence(config) → seqID Engine->>KV: allocate slot 0 Swift->>Engine: decodePrompt(seqID, tokens, count, offset) Engine->>KV: prefill prompt tokens loop Swift owns token budget (options.maxPredictionTokens) Swift->>Engine: sampleNext(seqID) Engine-->>Swift: SampleResult (piece / is_eos / was_cancelled) alt Task.isCancelled or EOS or argmax_eog Swift->>Swift: break loop end end Swift->>Engine: trimKV(seqID, promptTokenCount) Note over KV: KV restored to prompt-only state for next request Note over Swift,KV: On reset / abort Swift->>Engine: destroySequence(seqID) Engine->>KV: free slot 0 Swift->>Swift: "autocompleteSequenceID = -1"%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%% sequenceDiagram participant Swift as Swift (LlamaRuntimeCore) participant Engine as CotabbyInferenceEngine (C++) participant KV as llama.cpp KV Cache (slot 0) Note over Swift,KV: Single autocomplete sequence lifecycle Swift->>Swift: samplingConfig(from: options) — SamplingConfig() + field assign Swift->>Engine: createSequence(config) → seqID Engine->>KV: allocate slot 0 Swift->>Engine: decodePrompt(seqID, tokens, count, offset) Engine->>KV: prefill prompt tokens loop Swift owns token budget (options.maxPredictionTokens) Swift->>Engine: sampleNext(seqID) Engine-->>Swift: SampleResult (piece / is_eos / was_cancelled) alt Task.isCancelled or EOS or argmax_eog Swift->>Swift: break loop end end Swift->>Engine: trimKV(seqID, promptTokenCount) Note over KV: KV restored to prompt-only state for next request Note over Swift,KV: On reset / abort Swift->>Engine: destroySequence(seqID) Engine->>KV: free slot 0 Swift->>Swift: "autocompleteSequenceID = -1"Reviews (1): Last reviewed commit: "Update Cotabby for single-sequence infer..." | Re-trigger Greptile