Skip to content

feat: add LiteLLM as AI gateway provider#505

Open
RheagalFire wants to merge 2 commits into
srcbookdev:mainfrom
RheagalFire:feat/add-litellm-provider
Open

feat: add LiteLLM as AI gateway provider#505
RheagalFire wants to merge 2 commits into
srcbookdev:mainfrom
RheagalFire:feat/add-litellm-provider

Conversation

@RheagalFire
Copy link
Copy Markdown

Summary

  • Adds LiteLLM as a new AI provider, enabling users to route AI requests through 100+ LLM providers (OpenAI, Anthropic, Azure, Bedrock, Vertex AI, Groq, Ollama, etc.) via a single LiteLLM proxy.
  • Reuses the existing customApiKey and aiBaseUrl config fields, so no DB migration needed.

Motivation

LiteLLM acts as a unified AI gateway. Instead of configuring individual provider API keys, users can point srcbook at a single LiteLLM proxy that handles provider routing, load balancing, cost tracking, and fallbacks across 100+ providers. Useful for teams that already run a centralized LiteLLM proxy, or users who need providers not directly supported (e.g., AWS Bedrock, Vertex AI).

Changes

  • packages/shared/src/ai.mts - added LiteLLM: 'litellm' to AiProvider enum and gpt-4o-mini as default model
  • packages/api/ai/config.mts - added case 'litellm': that uses createOpenAI with proxy URL (default http://localhost:4000/v1), reuses customApiKey and aiBaseUrl fields
  • packages/web/src/routes/settings.tsx - added LiteLLM option to provider dropdown, added LiteLLM config panel with proxy URL + API key inputs and link to LiteLLM docs

Tests

Live E2E - AI SDK with LiteLLM proxy routing to Claude Sonnet 4.6:

$ node --input-type=module -e '...'

Model: claude-sonnet-4-6
Content: OK
Usage: {"promptTokens":13,"completionTokens":4,"totalTokens":17}
Finish reason: stop

This proves the full chain: createOpenAI (same SDK srcbook uses) -> LiteLLM proxy -> Claude Sonnet 4.6 -> response parsed.

Unit tests (6/6 pass):

$ bunx vitest run test/litellm-provider.test.mts

✓ test/litellm-provider.test.mts  (6 tests) 2ms
Test Files  1 passed (1)
Tests  6 passed (6)

Tests cover: provider enum registration, default model, provider validation, existing providers not broken, invalid provider rejection.

Regression (11/11 pass, 1 pre-existing failure in srcmd.test.mts unrelated to this PR):

$ bunx vitest run

✓ test/tsserver.test.mts  (3 tests)
✓ test/streaming-xml-parser.test.mts  (2 tests)
✓ test/litellm-provider.test.mts  (6 tests)
Tests  11 passed (11)

Risk / Compatibility

  • Additive only. Existing providers completely untouched.
  • No new dependencies added - uses the existing @ai-sdk/openai package.
  • No DB migration needed - reuses customApiKey and aiBaseUrl config fields.
  • Default proxy URL is http://localhost:4000/v1 (standard LiteLLM proxy port).

Example usage

In Settings, select "litellm" from the provider dropdown, then:

  1. Enter your LiteLLM proxy URL (e.g., http://localhost:4000/v1)
  2. Enter your LiteLLM API key
  3. Set model to any LiteLLM model ID (e.g., anthropic/claude-3-5-sonnet, openai/gpt-4o, bedrock/anthropic.claude-3-sonnet)

@RheagalFire
Copy link
Copy Markdown
Author

cc @nichochar

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant