Add com.codename1.ai package, ChatView, Speech/TTS, and build-time AI dependency injection#5035
Open
shai-almog wants to merge 8 commits into
Open
Add com.codename1.ai package, ChatView, Speech/TTS, and build-time AI dependency injection#5035shai-almog wants to merge 8 commits into
shai-almog wants to merge 8 commits into
Conversation
… dependency injection Introduces the full AI/LLM surface for Codename One plus the build-server plumbing that auto-wires every native dependency it needs. Unifies what was originally split into two PRs because the pieces never made sense on their own (the AI table refers to the speech/TTS class names; SimulatorRedirect consumes the JavaSEPort Ollama probe). ### com.codename1.ai - LlmClient with static factories for OpenAI, Anthropic, Gemini, Ollama, and a generic OpenAI-compatible endpoint. OpenAI is the only fully-implemented native client; it also drives Ollama / vLLM / llama.cpp because the wire format matches. Anthropic and Gemini compile and register, but throw a clear error pointing callers at the OpenAI-compat shim until their native clients land. - Streaming chat via a custom ConnectionRequest subclass that parses SSE line-by-line and dispatches deltas through Display.callSerially. AsyncResource.cancel() kills the socket. - Value types: ChatRequest (builder), ChatMessage, MessagePart (TextPart / ImagePart / ToolResultPart), Tool, ToolCall, ToolChoice, ResponseFormat, ChatResponse, Usage, Embedding + request/response, LlmException taxonomy. - ImageGenerator with OpenAI DALL-E (Replicate scaffold; on- device SD via the optional cn1-ai-stablediffusion cn1lib). - PromptTemplate, Tokenizer, RetryPolicy, ConversationStore, SafetyFilter. ### com.codename1.media.SpeechRecognizer + TextToSpeech - New core APIs routed through Display + CodenameOneImplementation no-op hooks. Platform ports override (iOS SFSpeechRecognizer / AVSpeechSynthesizer; Android android.speech.*). JavaSE ships a best-effort TTS implementation via say / espeak / SAPI; speech recognition stays unsupported unless cn1-ai-whisper is added. ### SecureStorage non-prompting overloads - Single-argument get/set/remove(account, ...) added next to the existing biometric-gated methods. For things like LLM API keys read on every network call where a biometric prompt would be unusable. Base class returns null/false on unsupported ports. ### com.codename1.components.ChatView - Scrollable message list (BoxLayout.Y_AXIS), ChatBubble + ChatInput components, theme-aware UIIDs, streaming appendToLastMessage that marshals through callSerially, and a bindToLlm(client, baseRequest) convenience that wires the input bar to chatStream and pipes deltas into the latest bubble. ### Build-time scanner additions - AiDependencyTable in the maven plugin: 18 entries mapping the com.codename1.ai.* class prefixes (plus the speech / TTS sister APIs) to iOS Pods, Swift Packages, Android Gradle deps, Info.plist usage strings, and Android permissions. - IPhoneBuilder + AndroidGradleBuilder pick up new branches inside their existing ASM class scanners; after the scan they apply matched entries. iOS deps route through the existing IOSDependencyManager so SPM-mode projects get SPM and Pods-mode projects get Pods automatically. Entries that bundle >2 GB native blobs (on-device Stable Diffusion) flip a cn1.ai.requiresBigUpload flag so the cloud build can abort pre-upload with a friendly "build locally" message. ### JavaSE simulator: Ollama detection - Probe at startup pings localhost:11434; sets cn1.ai.ollamaDetected when reachable. SimulatorRedirect reads the property and, with cn1.ai.simulatorRedirect=auto or =ollama, routes any LlmClient.openai(...) call through the local Ollama endpoint so unchanged production code can be debugged offline. ### Tests - 9 plugin tests (AiDependencyTableTest): pods, SPM routing, big-upload flagging, accumulator dedup, false-positive guard. - 23 core-unittests across 5 files: ChatRequestBuilderTest, JsonHelperTest (escaping, null omission, raw-JSON inlining, integer formatting), PromptTemplateTest, TokenizerTest, OpenAiSseDecoderTest (delta aggregation, fragmented tool-call argument reassembly, finish_reason capture, error status mapping including context_length_exceeded). ### scripts/create-ai-cn1lib.sh - Bootstrap helper that generates a new AI cn1lib repo from cn1lib-archetype and drops in a .github/workflows/publish.yml that publishes to Maven Central on every merge to master. ### Tracked follow-ups - iOS / Android native bridges for SpeechRecognizer + TextToSpeech. - BuildDaemon mirror of the scanner additions (separate PR). - Anthropic / Gemini native wire-format clients (today they route via OpenAI-compat shims). - The thirteen cn1-ai-* cn1lib repos themselves. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced May 24, 2026
Collaborator
Author
|
Compared 11 screenshots: 11 matched. |
Collaborator
Author
|
Compared 20 screenshots: 20 matched. |
Collaborator
Author
|
Compared 110 screenshots: 110 matched. Native Android coverage
✅ Native Android screenshot tests passed. Native Android coverage
Benchmark ResultsDetailed Performance Metrics
|
The static-analysis gate (.github/scripts/generate-quality-report.py) treats REC_CATCH_EXCEPTION, URF_UNREAD_FIELD, and UCF_USELESS_CONTROL_FLOW as hard-fail violations. Cleaned up the AI code accordingly: - OpenAiClient.postResponse + embed.postResponse: Split the bare `catch (Exception)` blocks into separate `catch (IOException)` + `catch (RuntimeException)` arms (Java 5 has no multi-catch). The IOException path covers JSON parse failures from JsonHelper.parseObject; the RuntimeException path covers ClassCastException / NPE while walking the response tree. Factored the EDT-dispatch of the resulting LlmException to a single shared `failParse` helper. - OpenAiClient.handleException: tightened the UTF-8 decoding catch from `Exception` to `UnsupportedEncodingException` (the only exception `new String(bytes, "UTF-8")` can actually throw, and even that is theoretical since UTF-8 is universally present). - OpenAiSseDecoder.mapErrorStatic: same split into IOException + RuntimeException catches. - ImageGenerator.postResponse + handleException: same split treatment. - ImageGenerator.ReplicateImageGenerator: dropped the unused apiKey field (URF_UNREAD_FIELD). The constructor parameter stays for API-shape stability when the real long-poll implementation lands. - ChatBubble: removed the empty `initComponent()` override (UCF_USELESS_CONTROL_FLOW). The framework consults UIManager on attach anyway, so the override added no behavior. Verified via local `mvn install -Plocal-dev-javase` against the core-unittests SpotBugs step: 0 forbidden violations remain. The 2 leftover findings (RCN_REDUNDANT_NULLCHECK_OF_NONNULL_VALUE and WMI_WRONG_MAP_ITERATOR) are style-level and not in the gate's forbidden set. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Collaborator
Author
|
Compared 110 screenshots: 110 matched. Benchmark Results
Build and Run Timing
Detailed Performance Metrics
|
Collaborator
Author
|
Compared 109 screenshots: 109 matched. Benchmark Results
Build and Run Timing
Detailed Performance Metrics
|
The CI static-analysis gate also enforces RCN_REDUNDANT_NULLCHECK_OF_NONNULL_VALUE (line 841 of .github/scripts/generate-quality-report.py), which I missed when the previous round only checked the *_WOULD_HAVE_BEEN_A_NPE sibling. Util.readInputStream is contractually non-null on success, so the `body == null` check after `byte[] body = Util.readInputStream(input)` is dead code. Removed it. Verified locally: 0 forbidden bug instances remain in the spotbugsXml.xml output; the single remaining finding (WMI_WRONG_MAP_ITERATOR in JsonHelper.writeValue) is a style-level performance hint not in the gate's forbidden set. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Build-test (8) runs PMD via maven-pmd-plugin in core-unittests and feeds every violation through the same forbidden-rule gate the SpotBugs path uses. Cleaned up 87 violations across nine rule categories. Verified locally with mvn verify in core-unittests: zero violations. - MissingOverride (~50 instances) -- added @OverRide to every anonymous Runnable / SuccessCallback / StreamingListener / ActionListener and every overridden provider method (getProvider / chat / chatStream / embed / consume / finish / mapError / generate / actionPerformed / onSucess / etc.) across AnthropicClient, GeminiClient, OpenAiClient, OpenAiSseDecoder, ImageGenerator, SafetyFilter, StreamingListener, StreamingChatRequest, ChatBubble, ChatInput, ChatView, RecognitionCallback, CodenameOneImplementation. - ForLoopCanBeForeach (5 instances) -- converted classic index-counted loops to enhanced for in ChatMessage, ConversationStore, and OpenAiSseDecoder. The tool-call delta loop in OpenAiSseDecoder uses an external counter to keep `i` as the default value for missing `index` fields. - ControlStatementBraces (~12 instances) -- added braces to single-line if/else in OpenAiClient (body-builder), ImageGenerator, ChatBubble.defaultUiidFor, and ConversationStore.parseRole. - AvoidStringBufferField (2 instances) -- @SuppressWarnings on the short-lived `content` and `arguments` accumulators in OpenAiSseDecoder. The decoder lives for one SSE stream so the memory-leak concern the rule guards against doesn't apply. - EmptyCatchBlock (1) -- added Log.e on the listener.onError swallow in StreamingChatRequest.failWith. - SimplifyBooleanReturns (1) -- collapsed the trailing `if (...) return true; return false;` in RetryPolicy.shouldRetry to a direct `return t instanceof LlmNetworkException`. - UnnecessaryFullyQualifiedName (4) -- replaced inline `java.io.IOException`, `java.io.UnsupportedEncodingException`, `com.codename1.ui.Display.getInstance()` with their imported short forms across OpenAiClient, ImageGenerator, CodenameOneImplementation. - UnnecessaryModifier (2) -- dropped `public static` from the nested Adapter classes in StreamingListener and RecognitionCallback (implicit on interface members in Java). - UnusedFormalParameter (1) -- @SuppressWarnings on ImageGenerator.ReplicateImageGenerator's apiKey constructor parameter (kept for API-shape stability when the real long-poll implementation lands). Also removed an obsolete Arrays.class workaround in ConversationStore that was a relic of an earlier import-warning fight. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
build-test (8) also runs Checkstyle (build-test (17) ran PMD but not Checkstyle in the same path -- both gates must pass). The 8 IndentationCheck failures all flagged the leading `+` on multi-line string concatenations being two columns short of the expected continuation indent. Indented the continuation lines by 2 extra spaces in the multi-line UnsupportedOperationException messages in: - AnthropicClient.chat / .embed - GeminiClient.chat / .embed - ImageGenerator.onDevice / .ReplicateImageGenerator.generate Verified locally via `mvn checkstyle:check`: 0 violations. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
✅ Continuous Quality ReportTest & Coverage
Static Analysis
Generated automatically by the PR CI workflow. |
Acts on six pieces of feedback raised on PR #5035: 1. **Drop scripts/create-ai-cn1lib.sh** -- redundant now that the cn1libs ship in-tree. 2. **Fold JsonHelper into JSONParser**. The serialize / RawJson / asMap / asList / getString / getInt / getDouble surface (plus convenience parseJSON(byte[]) / parseJSON(String) overloads) is now public API on com.codename1.io.JSONParser. JsonHelper and its test-only bridge are deleted; ImageGenerator, OpenAiClient, OpenAiSseDecoder, ConversationStore all import JSONParser instead. 3. **Endpoint URLs hoisted to named constants** on LlmClient (DEFAULT_OPENAI_URL, DEFAULT_ANTHROPIC_URL, DEFAULT_GEMINI_URL, DEFAULT_OLLAMA_URL). Gemini stays on /v1beta -- that's still the only path that exposes streaming generateContent + tool calls; the constants make the version choice grep-visible and easy to pin. setBaseUrl(...) is the customisation hook for self-hosted gateways and regional endpoints. 4. **LlmException.ErrorType enum** so callers can do a single try/catch + switch (the more idiomatic shape for most apps): switch (e.getType()) { case RATE_LIMIT: ... case AUTH: ... case CONTEXT_LENGTH: ... ... } The subclasses (LlmRateLimitException etc.) remain for cases where instanceof + extra typed state (e.g. getRetryAfterSeconds) is more ergonomic, but the class javadoc on LlmException recommends the enum-switch form. Each subclass now passes its matching ErrorType up the constructor chain. 5. **Tool / ToolCall linkage**. Tool gains an optional ToolHandler constructor parameter; ToolCall gains execute(List<Tool>) + findTool(List<Tool>) so callers can dispatch the model's tool calls through the matching Tool without sprinkling name-based lookups across application code. Apps that only want the description-shape can still construct Tools without a handler. 6. **ChatView decoupled from LlmClient**. Moved bindToLlm into a new com.codename1.ai.LlmChatBinding helper class. ChatView is now a pure messaging UI -- it consumes ChatMessage (the same data envelope, but neutral enough to model peer-to-peer conversations: USER = self, ASSISTANT = peer, SYSTEM = system notice) and emits send/attach/voice events to listeners. The class javadoc shows a WhatsApp-style binding alongside the LlmChatBinding example. Plus **13 AI cn1lib scaffolds** under maven/cn1-ai-<feature>/: cn1-ai-mlkit-text / -barcode / -face / -labeling / -translate / -smartreply / -langid / -pose / -segmentation / -docscan, cn1-ai-tflite, cn1-ai-whisper, cn1-ai-stablediffusion Each ships a public Java facade in the corresponding com.codename1.ai.<feature> package, throws UnsupportedOperationException on every method until per-platform native bridges land in follow-up commits, and is recognised by the build-server's AiDependencyTable so iOS Pods / Swift Packages / Android Gradle deps / Info.plist usage strings / Android permissions still wire up automatically the moment an app imports a class. Each module is added to the maven aggregator reactor and builds clean today (`mvn verify` from core-unittests passes 2587 tests). The cn1libs are intentionally single-Java-module scaffolds rather than the full 7-module common/ios/android/javase/javascript/win/lib archetype layout; expanding to the full layout is per-cn1lib work that lands when each platform's native code is ready. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
build-test (8) flagged three java.io.IOException / .InputStreamReader / .ByteArrayInputStream FQNs inside the new parseJSON(byte[]) and parseJSON(String) helpers. Added the missing imports and switched to short names. Local mvn verify is clean (2587 tests, 0 PMD). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…e pattern
Acts on two follow-up review notes:
1. The seven LlmException subclasses (Auth / RateLimit /
InvalidRequest / ContextLength / ModelOverloaded / Server /
Network) are redundant now that LlmException carries the
ErrorType enum. Deleted all seven; moved retryAfterSeconds (the
only piece of state that lived only on the rate-limit subclass)
onto LlmException with a getter that returns -1 when not
applicable. Every existing caller and test was rewritten to:
new LlmException(message, status, code, body, cause,
LlmException.ErrorType.X)
and instanceof checks were turned into:
e.getType() == LlmException.ErrorType.X
The class javadoc now shows a single try/catch + switch over
getType() as the recommended pattern.
2. The 13 cn1lib scaffolds no longer throw UnsupportedOperationException
synchronously. Each facade now uses the standard Codename One
NativeInterface pattern:
- A package-private Native<Feature> interface extending
NativeInterface defines the platform contract.
- The facade resolves it via NativeLookup.create(Native<Feature>.class)
in a background thread (Display.scheduleBackgroundTask) and
completes an AsyncResource on the EDT.
- When no platform impl is registered the facade still completes
gracefully -- the AsyncResource fires error() with a clear
LlmException explaining the platform isn't wired up yet --
instead of throwing. App code can adopt the API today and the
platform bridges (iOS Obj-C / Android Java) fill in transparently
as they ship per-cn1lib in follow-up commits.
Static isSupported() on each facade lets UI code gate behaviour
without having to call the method and inspect the error.
Versioning + release alignment (review note 3):
Each cn1lib pom inherits <parent> codenameone:8.0-SNAPSHOT, so
the cn1libs follow the core's version through update-version.sh
automatically and land on Maven Central as part of every
Codename One release. The dependency on codenameone-core is
declared <scope>provided</scope> so the cn1lib's published pom
doesn't transitively force a core version on consumers -- they
keep whatever cn1.version their app already declares.
Verified locally with `mvn clean verify -Plocal-dev-javase`:
2587 tests, 0 failures, 0 errors, 0 forbidden PMD / SpotBugs /
Checkstyle violations.
Native platform implementations for the 13 cn1libs (calling
GoogleMLKit / TensorFlowLiteSwift / libwhisper.a / Core ML / ONNX
Runtime on iOS and Android) require device-testable bindings and
ship in follow-up per-cn1lib commits.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Introduces the full AI/LLM surface for Codename One plus the
build-server plumbing that auto-wires every native dependency it
needs. Unifies what was originally split into two PRs (#5033, #5034)
because the pieces never made sense on their own (the AI table
refers to the speech/TTS class names;
SimulatorRedirectconsumesthe
JavaSEPortOllama probe). Rebased on current master.com.codename1.aiLlmClientstatic factories for OpenAI, Anthropic, Gemini,Ollama, and a generic OpenAI-compatible endpoint. OpenAI is the
only fully-implemented native client; it also drives Ollama /
vLLM / llama.cpp because the wire format matches. Anthropic and
Gemini compile and register, but throw a clear error pointing
callers at the OpenAI-compat shim until their native clients
land.
ConnectionRequestsubclassthat parses SSE line-by-line and dispatches deltas through
Display.callSerially.AsyncResource.cancel()kills thesocket.
ChatRequest(builder),ChatMessage,MessageParthierarchy,Tool,ToolCall,ToolChoice,ResponseFormat,ChatResponse,Usage,Embedding+request/response, full
LlmExceptiontaxonomy.ImageGeneratorwith OpenAI DALL-E (Replicate scaffold;on-device SD via the optional
cn1-ai-stablediffusioncn1lib).PromptTemplate,Tokenizer,RetryPolicy,ConversationStore,SafetyFilter.com.codename1.media.SpeechRecognizer+TextToSpeechNew core APIs routed through
Displayand no-opCodenameOneImplementationhooks. Platform ports will override(iOS
SFSpeechRecognizer/AVSpeechSynthesizer; Androidandroid.speech.*).JavaSEPortships a best-effort TTS viasay/espeak/ SAPI; speech recognition stays unsupportedunless
cn1-ai-whisperis added.SecureStoragenon-prompting overloadsSingle-arg
get/set/remove(account, ...)added next to theexisting biometric-gated methods. For things like LLM API keys
read on every network call where a biometric prompt would be
unusable. Base class returns
null/falseon unsupported ports.com.codename1.components.ChatViewScrollable message list,
ChatBubble+ChatInput, theme-awareUIIDs, streaming
appendToLastMessagethat marshals throughcallSerially, and abindToLlm(client, baseRequest)convenience that wires the input bar directly to
chatStream.Build-time scanner additions
AiDependencyTablein the maven plugin: 18 entries mappingcom/codename1/ai/*(plus the speech/TTS sister APIs) to iOSPods, Swift Packages, Android Gradle deps,
Info.plistusagestrings, and Android permissions.
IPhoneBuilder+AndroidGradleBuilderpick up newbranches inside their existing ASM class scanners. After the
scan they apply matched entries; iOS deps route through the
existing
IOSDependencyManagerso SPM-mode projects get SPMand Pods-mode projects get Pods automatically. Entries
bundling >2 GB native blobs (on-device Stable Diffusion) flip
a
cn1.ai.requiresBigUploadflag so the cloud build can abortpre-upload with a friendly "build locally" message.
JavaSE simulator: Ollama detection
probeOllamaAsync()pingslocalhost:11434at startup; setscn1.ai.ollamaDetectedwhen reachable.SimulatorRedirectreads that property and, with
cn1.ai.simulatorRedirect=autoor
=ollama, routes anyLlmClient.openai(...)call throughthe local Ollama endpoint so unchanged production code can be
debugged offline without API charges.
Tests
AiDependencyTableTest): pods, SPMrouting, big-upload flagging, accumulator dedup,
false-positive guard.
ChatRequestBuilderTest,JsonHelperTest(escaping, null omission, raw-JSON inlining,integer formatting),
PromptTemplateTest,TokenizerTest,OpenAiSseDecoderTest(delta aggregation, fragmentedtool-call argument reassembly, terminal
finish_reasoncapture, error status mapping including
context_length_exceeded).mvn install -pl core,codenameone-maven-plugin,javase -am -Plocal-dev-javase -DskipTests -Dspotbugs.skip=truebuilds clean.scripts/create-ai-cn1lib.shBootstrap helper that generates a new AI cn1lib repo from
cn1lib-archetypeand drops in a.github/workflows/publish.ymlthat publishes to Maven Central on every merge to master.
Tracked follow-ups (not blocking)
SpeechRecognizer+TextToSpeech(need device testing).cn1-ai-*cn1lib repos themselves (bootstrap via the new script).Speech.framework/AVFAudio/CoreMLetc. — most arrive via their accompanying pods; standalone framework injection pending.Test plan
cd maven && mvn install -pl core,codenameone-maven-plugin,javase -am -Plocal-dev-javase -DskipTests -Dspotbugs.skip=truebuilds cleancd maven/codenameone-maven-plugin && mvn test -Dtest=AiDependencyTableTest-- 9/9 passcd maven/core-unittests && mvn test -Dtest="ChatRequestBuilderTest,JsonHelperTest,PromptTemplateTest,TokenizerTest,OpenAiSseDecoderTest"-- 23/23 passLlmClient.openai(System.getenv("OPENAI_API_KEY")).chatStream(req, listener)streams tokenscn1.ai.simulatorRedirect=ollamaroutesLlmClient.openai("sk-fake")calls tolocalhost:11434TextToSpeech.speak("hello")works on the simulator (macOS viasay)android.xpermissionswith CAMERA + using a (future)com.codename1.ai.mlkit.barcode.*class does not produce duplicate<uses-permission>lines in the final manifestGenerated with Claude Code