LLM Gateway

LLM Gateway is an open-source middleware layer between applications and various Large Language Model providers, consolidating multiple AI services into a single entry point to simplify the integration of generative AI into software workflows.

The software is deployed as a server and provides a web-based dashboard for management. It allows users to route requests to different providers while maintaining a consistent API format, which reduces the need to rewrite integration code when switching models or adding new providers to a project. Users can deploy the system on their own infrastructure using Docker for complete control over data and configuration, or use a hosted version for immediate access.

Key features

• Unified API interface compatible with the OpenAI API format
• Multi-provider support for OpenAI, Anthropic, and Google Vertex AI
• Centralized API key management for different LLM providers
• Usage analytics for tracking requests and token consumption
• Cost tracking across all LLM interactions
• Performance monitoring to compare model response times
• Self-hosted deployment option via Docker
• Integrated LLM playground for testing prompts

The system is built with a TypeScript stack, utilizing a Hono backend and Next.js for the frontend dashboard and playground. It employs Drizzle ORM for database management and uses PostgreSQL and Redis for data storage and caching. The architecture separates the API gateway logic from the administrative UI, internal management tools, and documentation site. The project follows a monorepo structure, separating the gateway, API, and various frontend applications into distinct packages.

Core functionality is licensed under AGPLv3, while additional enterprise features include advanced billing, subscription management, and extended data retention. LLM Gateway is a management layer for organizations that require observability and centralized control over their AI inference costs and provider dependencies.