⚒️ LLM-Forge

A unified response parser and SDK generator for LLM APIs across multiple programming languages

LLM-Forge provides a production-ready, type-safe way to parse and normalize responses from multiple LLM providers (OpenAI, Anthropic, Cohere, Google AI, Mistral, and more) with support for generating client libraries in 6 languages: TypeScript, Python, Rust, Go, Java, and C#.

Why LLM-Forge?

🔄 Unified API - One interface for 12+ LLM providers
🚀 Ultra-Fast - 136K-454K ops/sec parsing performance
🛡️ Type-Safe - Full TypeScript inference and validation
🧪 Battle-Tested - 666 tests, 93.77% coverage
📦 Zero Dependencies - Lightweight, production-ready
🌐 Multi-Language - Generate SDKs in 6 languages

✨ Features

Provider Support (12 Providers)

✅ Multi-Provider Parsing: Unified response format for 12 LLM providers
✅ Auto-Detection: Automatically detect provider from response structure
✅ Streaming Support: Real-time streaming chunk parsing
✅ Type-Safe: Full TypeScript type inference and safety
✅ Production Ready: 93.77% test coverage, 666 passing tests
✅ High Performance: 136K-454K ops/sec parsing, 1-10M ops/sec detection

Code Generation (6 Languages)

✅ TypeScript: Full type inference, decorators, async/await
✅ Python: Type hints, Pydantic models, async support
✅ Rust: Serde, strong typing, Result<T,E>
✅ Java: Record classes, Jackson, CompletableFuture
✅ C#: Record types, System.Text.Json, async streams
✅ Go: Struct tags, JSON marshaling, context support

Enterprise Features

✅ CI/CD Pipeline: 7 GitHub Actions workflows for automation
✅ Security Scanning: Multi-layer security with CodeQL, npm audit, OSSF
✅ Performance Monitoring: Automated benchmarking and regression detection
✅ Automated Releases: npm and GitHub Packages publishing
✅ Comprehensive Documentation: Production guides and API docs

📊 Status

Production Ready ✅

Test Coverage:  93.77% ✅
Tests Passing:  666/666 (100%) ✅
Benchmarks:     27 performance tests ✅
CI/CD:          7 automated workflows ✅
Documentation:  Complete ✅

🚀 Quick Start

Installation

# Using npm
npm install @llm-dev-ops/llm-forge

# Using yarn
yarn add @llm-dev-ops/llm-forge

# Using pnpm
pnpm add @llm-dev-ops/llm-forge

Basic Usage - Response Parsing

import { parseResponse } from '@llm-dev-ops/llm-forge';

// Parse any LLM provider response - automatically detects the provider
const response = await fetch('https://api.openai.com/v1/chat/completions', {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${process.env.OPENAI_API_KEY}`,
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    model: 'gpt-4',
    messages: [{ role: 'user', content: 'Hello, world!' }]
  })
});

const data = await response.json();
const parsed = await parseResponse(data);

if (parsed.success) {
  console.log(parsed.response.messages[0].content);
  // Output: "Hello! How can I help you today?"

  console.log(`Provider: ${parsed.response.provider}`);
  // Output: "Provider: openai"

  console.log(`Model: ${parsed.response.model.id}`);
  // Output: "Model: gpt-4"

  console.log(`Tokens: ${parsed.response.usage.totalTokens}`);
  // Output: "Tokens: 25"
}

Real-World Example - Multi-Provider Chat

import { parseResponse } from '@llm-dev-ops/llm-forge';

async function chat(provider: 'openai' | 'anthropic' | 'cohere', message: string) {
  // Use any provider's API - LLM-Forge normalizes the response
  const endpoints = {
    openai: 'https://api.openai.com/v1/chat/completions',
    anthropic: 'https://api.anthropic.com/v1/messages',
    cohere: 'https://api.cohere.ai/v1/chat'
  };

  const response = await fetch(endpoints[provider], {
    method: 'POST',
    headers: { /* provider-specific headers */ },
    body: JSON.stringify({ /* provider-specific payload */ })
  });

  const data = await response.json();
  const parsed = await parseResponse(data);

  if (parsed.success) {
    return {
      text: parsed.response.messages[0].content,
      tokens: parsed.response.usage.totalTokens,
      cost: calculateCost(parsed.response.usage, parsed.response.model.id)
    };
  }

  throw new Error(parsed.error.message);
}

// Works seamlessly with any provider
const openAIResult = await chat('openai', 'Explain quantum computing');
const claudeResult = await chat('anthropic', 'Explain quantum computing');
const cohereResult = await chat('cohere', 'Explain quantum computing');

Auto-Detection

import { parseResponse } from '@llm-dev-ops/llm-forge';

// Automatically detects provider from response structure
const openAIResponse = await parseResponse(openAIData);    // Detects OpenAI
const anthropicResponse = await parseResponse(claudeData); // Detects Anthropic
const cohereResponse = await parseResponse(cohereData);    // Detects Cohere

💡 Use Cases

1. Multi-Provider Chatbots

Build chatbots that can seamlessly switch between different LLM providers based on cost, performance, or availability.

2. LLM Abstraction Layer

Create a unified interface for your application that works with any LLM provider, making it easy to switch providers or A/B test different models.

3. Cost Optimization

Parse usage data from multiple providers to track and optimize token usage and costs across your organization.

4. SDK Generation

Generate type-safe client libraries in multiple programming languages from OpenAPI specifications, accelerating development across teams.

5. LLM Gateway/Proxy

Build an LLM gateway that routes requests to different providers, normalizes responses, and provides unified monitoring and analytics.

6. Testing & Development

Easily switch between providers during development and testing without changing your application code.

Provider-Specific Parsing

import { OpenAIProvider, AnthropicProvider } from '@llm-dev-ops/llm-forge';

const openai = new OpenAIProvider();
const result = await openai.parse(openAIResponse);

const anthropic = new AnthropicProvider();
const claudeResult = await anthropic.parse(anthropicResponse);

Streaming Support

import { OpenAIProvider } from '@llm-dev-ops/llm-forge';

const provider = new OpenAIProvider();

// Parse streaming chunks
for await (const chunk of streamingResponse) {
  const parsed = await provider.parseStream(chunk);
  if (parsed.success) {
    process.stdout.write(parsed.response.messages[0].content);
  }
}

🎯 Supported Providers

Provider	Status	Detection	Parsing	Streaming
OpenAI	✅ Complete	✅	✅	✅
Anthropic	✅ Complete	✅	✅	✅
Google AI	✅ Complete	✅	✅	✅
Cohere	✅ Complete	✅	✅	✅
Mistral	✅ Complete	✅	✅	✅
Azure OpenAI	✅ Complete	✅	✅	✅
Hugging Face	✅ Complete	✅	✅	⚠️ Limited
Replicate	✅ Complete	✅	✅	⚠️ Limited
Together AI	✅ Complete	✅	✅	⚠️ Limited
Perplexity	✅ Complete	✅	✅	✅
OpenRouter	✅ Complete	✅	✅	✅
Custom	✅ Complete	✅	✅	⚠️ Provider-dependent

🔧 Code Generation

Generate TypeScript Client

import { generateTypeScript } from '@llm-dev-ops/llm-forge';

const schema = {
  name: 'ChatCompletion',
  properties: {
    messages: { type: 'array', items: { type: 'Message' } },
    model: { type: 'string' }
  }
};

const code = await generateTypeScript(schema);
console.log(code);

Supported Languages

Language	Status	Package Manager	Type Safety	Async Support
TypeScript	✅ Complete	npm	Full	async/await
Python	✅ Complete	pip	Type hints	async/await
Rust	✅ Complete	cargo	Strong	tokio
Java	✅ Complete	Maven/Gradle	Strong	CompletableFuture
C#	✅ Complete	NuGet	Strong	async/await
Go	✅ Complete	go modules	Static	goroutines

📈 Performance

Benchmarks (ops/sec)

Provider Detection:

OpenAI: 9.7M ops/sec
Anthropic: 9.4M ops/sec
Cohere: 8.7M ops/sec
Mistral: 6.7M ops/sec
Google AI: 5.5M ops/sec

Response Parsing:

Mistral: 454K ops/sec (fastest)
OpenAI: 422K ops/sec
Anthropic: 368K ops/sec
Cohere: 313K ops/sec
Google AI: 137K ops/sec

Streaming:

OpenAI: 504K chunks/sec
Anthropic: 485K chunks/sec

Benchmarked on Node.js 20 with Vitest bench suite (27 benchmarks)

🏗️ Architecture

LLM-Forge uses a layered architecture:

┌─────────────────────────────────────────────────────┐
│  Provider Responses (OpenAI, Anthropic, etc.)       │
└────────────────┬────────────────────────────────────┘
                 │
┌────────────────▼────────────────────────────────────┐
│  Provider Detection & Auto-detection                │
└────────────────┬────────────────────────────────────┘
                 │
┌────────────────▼────────────────────────────────────┐
│  Unified Response Parsing                           │
│  - Message extraction                               │
│  - Metadata normalization                           │
│  - Token usage tracking                             │
│  - Error handling                                   │
└────────────────┬────────────────────────────────────┘
                 │
┌────────────────▼────────────────────────────────────┐
│  Code Generation (6 languages)                      │
│  - Type generation                                  │
│  - Client generation                                │
│  - Serialization                                    │
└─────────────────────────────────────────────────────┘

See docs/ARCHITECTURE.md for detailed architecture documentation.

🧪 Testing

Test Coverage

Overall Coverage:    93.77%
Providers Coverage:  92.68%
Generators Coverage: 98.17%
Parsers Coverage:    98.04%
Core Coverage:       97.73%

Total Tests: 666 passing
Test Files:  23 files
Duration:    ~10 seconds

Run Tests

# Run all tests
npm test

# Run with coverage
npm run test:coverage

# Run benchmarks
npm run bench

# Run specific test file
npm test tests/providers/integration.test.ts

🔒 Security

LLM-Forge implements multiple security layers:

✅ Daily Security Scans: Automated vulnerability detection
✅ CodeQL Analysis: Static security analysis
✅ Secret Detection: TruffleHog scanning
✅ License Compliance: Automated license checking
✅ Dependency Updates: Dependabot automation
✅ OSSF Scorecard: Security best practices validation

See docs/CI_CD_PIPELINE.md for security documentation.

🔄 CI/CD Pipeline

7 automated workflows ensure quality:

PR Validation - Quality gates for pull requests
Continuous Integration - Multi-OS testing (Ubuntu, macOS, Windows)
Security Scanning - Multi-layer security analysis
Performance Monitoring - Benchmark tracking and regression detection
Release & Publish - Automated npm publishing
Dependabot Auto-Merge - Safe dependency updates
Stale Management - Issue/PR lifecycle management

See .github/README.md for workflow documentation.

📚 Documentation

User Guides

Production Readiness - Deployment guide
CI/CD Pipeline - Pipeline documentation
Architecture - System architecture

Implementation

Provider System - Provider implementation
Template Engine - Code generation templates
Language Strategy - Multi-language support

Reference

Implementation Summary - Complete implementation details
DevOps Deliverables - CI/CD and deployment details
Multi-Language Success - Code generation implementation

🛠️ Development

Prerequisites

Node.js >= 20.0.0
npm >= 10.0.0
TypeScript >= 5.3.3

Setup

# Clone repository
git clone https://github.com/globalbusinessadvisors/llm-forge.git
cd llm-forge

# Install dependencies
npm install

# Run tests
npm test

# Build project
npm run build

# Run development mode
npm run dev

# Run benchmarks
npm run bench

# Run all quality checks
npm run quality

CLI Usage

LLM-Forge includes a command-line interface for generating SDKs:

# Run CLI in development
npm run cli -- --help

# Generate SDK from OpenAPI spec
npm run cli -- generate --input openapi.json --output ./sdk --language typescript

# After installation (globally or in project)
npx llm-forge generate --input openapi.json --language python

Project Structure

llm-forge/
├── src/
│   ├── core/           # Template engine and type system
│   ├── generators/     # Language-specific code generators
│   ├── parsers/        # OpenAPI and Anthropic parsers
│   ├── providers/      # Provider-specific parsers (12 providers)
│   ├── schema/         # Schema validation
│   └── types/          # TypeScript type definitions
├── tests/
│   ├── core/           # Core functionality tests
│   ├── generators/     # Code generator tests
│   ├── parsers/        # Parser tests
│   ├── providers/      # Provider tests (integration, benchmarks)
│   └── schema/         # Schema validation tests
├── docs/               # Comprehensive documentation
├── examples/           # Example usage
├── scripts/            # Build and utility scripts
└── .github/
    ├── workflows/      # 7 CI/CD workflows
    └── dependabot.yml  # Dependency automation

Available Scripts

npm test              # Run all tests
npm run test:coverage # Run tests with coverage report
npm run bench         # Run performance benchmarks
npm run type-check    # TypeScript type checking
npm run lint          # ESLint code linting
npm run format        # Prettier code formatting
npm run build         # Build package
npm run clean         # Clean build artifacts
npm run quality       # Run all quality checks

🤝 Contributing

We welcome contributions! Please follow these steps:

Fork the repository
Create a feature branch (git checkout -b feature/amazing-feature)
Make your changes with tests
Run quality checks (npm run quality)
Commit your changes (git commit -m 'feat: add amazing feature')
Push to the branch (git push origin feature/amazing-feature)
Open a Pull Request

All PRs must pass:

✅ TypeScript type checking
✅ ESLint linting
✅ Prettier formatting
✅ All 666 tests
✅ 93%+ code coverage
✅ Security scans

See docs/CI_CD_PIPELINE.md for detailed contribution guidelines.

📦 Publishing

npm

npm install @llm-dev-ops/llm-forge

GitHub Packages

npm install @llm-dev-ops/llm-forge

🗺️ Roadmap

✅ Phase 1: Foundation (Complete)

✅ Provider response parsing (12 providers)
✅ Unified response format
✅ Auto-detection system
✅ Streaming support

✅ Phase 2: Code Generation (Complete)

✅ TypeScript generator
✅ Python generator
✅ Rust generator
✅ Java generator
✅ C# generator
✅ Go generator

✅ Phase 3: Production Ready (Complete)

✅ Comprehensive testing (666 tests)
✅ 93.77% code coverage
✅ Performance benchmarking
✅ CI/CD pipeline (7 workflows)
✅ Security scanning
✅ Complete documentation

🔮 Phase 4: Future Enhancements (Planned)

📄 License

Apache License 2.0 - see LICENSE for details.

🙏 Acknowledgments

Built with enterprise-grade quality using:

Testing: Vitest
CI/CD: GitHub Actions
Security: CodeQL, TruffleHug, OSSF Scorecard
Coverage: Codecov
Type Safety: TypeScript

📊 Project Metrics

Lines of Code:       ~15,000
Test Coverage:       93.77%
Tests:              666 passing
Benchmarks:         27 performance tests
Providers:          12 supported
Languages:          6 code generators
CI/CD Workflows:    7 automated
Documentation:      35+ comprehensive docs
Performance:        136K-454K ops/sec parsing
Security:           Multi-layer scanning

🆘 Support

Documentation: docs/
Issues: GitHub Issues
Discussions: GitHub Discussions
CI/CD Status: GitHub Actions
npm Package: npmjs.com
Coverage: Codecov

Status: ✅ Production Ready | License: Apache 2.0 | Version: 1.0.0

Made with ❤️ by the LLM-Dev-Ops Team

Getting Started · Documentation · Report Bug · Request Feature

Name		Name	Last commit message	Last commit date
Latest commit History 33 Commits
.claude		.claude
.github		.github
.swarm		.swarm
agents		agents
deploy		deploy
docs		docs
examples		examples
forge-benchmarks		forge-benchmarks
plans		plans
schemas		schemas
scripts		scripts
src		src
templates		templates
tests		tests
.eslintrc.json		.eslintrc.json
.gcloudignore		.gcloudignore
.gitignore		.gitignore
.mcp.json		.mcp.json
.prettierignore		.prettierignore
.prettierrc.json		.prettierrc.json
CLAUDE.md		CLAUDE.md
LICENSE		LICENSE
README.md		README.md
package-lock.json		package-lock.json
package.json		package.json
tsconfig.json		tsconfig.json
tsconfig.test.json		tsconfig.test.json
tsup.config.ts		tsup.config.ts
vitest.config.ts		vitest.config.ts

Uh oh!

Folders and files

Latest commit

History

Repository files navigation