Portkey AI Gateway

Portkey AI Gateway is an open-source web app and server that provides a unified API for routing requests to over 1,600 language, vision, audio, and image models. It acts as a lightweight intermediary layer between applications and various LLM providers to improve reliability, security, and observability. This tool allows developers to integrate with any language model in a short timeframe without rewriting their entire integration logic.

The software is deployed as a server and can be run locally via Node.js, through Docker, or hosted on platforms like Cloudflare and Replit. It supports integration with multiple SDKs and libraries, including Python, JavaScript, LangChain, and LlamaIndex. By using a single API signature, it enables users to switch between providers without changing the core application code.

Key features

• Automatic retries with exponential backoff to prevent network overload
• Fallback routing to alternative providers or models upon request failure
• Load balancing across multiple API keys or providers using weighted distribution
• Input and output guardrails to ensure compliance and security standards
• Multi-modal support for vision, audio, and image generation models
• MCP Gateway for centralized authentication and observability of MCP servers
• Request timeouts to terminate calls that exceed a specified duration
• Smart caching of responses to reduce costs and latency

The architecture includes a Gateway Console for viewing local logs and a configuration system for defining routing rules. It integrates with agent frameworks such as AutoGen, CrewAI, and Phidata to support complex agentic workflows. For organizational needs, it offers a Model Context Protocol (MCP) gateway that provides a single authentication layer, access control, and identity forwarding for MCP-compatible clients.

This tool is positioned as an enterprise-ready routing layer for developers building LLM-based applications who require high availability, cost management, and centralized control over multiple AI providers.