OpenLIT

OpenLIT is an open-source web app and server that provides observability for Generative AI and LLM applications. It enables developers to monitor AI application health, track costs, and manage prompts through a web-based dashboard and server. The software is deployed via Docker or Kubernetes and integrates into applications using SDKs or a CLI.

The platform uses OpenTelemetry standards to collect traces and metrics from LLMs, vector databases, and GPUs. It supports a wide range of AI frameworks and providers, allowing developers to move from experimental testing to production with full-stack monitoring. A dedicated CLI also provides observability for local coding agents by installing vendor hooks that emit traces for sessions, tool calls, and file edits.

Key features

• OpenTelemetry-native SDKs for Python, TypeScript, and Go
• Automated LLM-as-a-Judge evaluations for hallucination, bias, and toxicity
• Prompt Hub for versioning and managing prompts across applications
• Analytics dashboards for tracking performance, costs, and user interactions
• Rule Engine for matching runtime trace attributes with conditional logic
• Centralized API key and secrets management
• OpenGround playground for side-by-side LLM comparison
• Fleet Hub for managing OpenTelemetry Collectors via OpAMP
• Cost tracking for custom and fine-tuned models

OpenLIT is built on a stack that includes ClickHouse for data storage and supports over 50 integrations, including LangChain, LlamaIndex, OpenAI, and various vector databases like Pinecone, ChromaDB, and Milvus. It is designed for AI engineers who require vendor-neutral observability and the ability to export data to other OTLP-compatible tools. The architecture follows Semantic Conventions with the OpenTelemetry community to ensure alignment with industry standards for distributed tracing.

It serves as a comprehensive LLMOps infrastructure tool for monitoring, evaluating, and optimizing generative AI workflows.