Open Source Alternatives to Datadog LLM Observability

An observability platform for AI agents that provides tools to evaluate, improve, and trace agent behavior in both offline experimentation and production environments. It allows AI engineers and data scientists to create versioned golden datasets from production traces, run experiments across different prompts and models, and use automated or human evaluators to measure quality, cost, and latency. The product integrates agent tracing with backend services, infrastructure signals, and user sessions to help teams pinpoint failures and monitor for hallucinations or prompt injections. It is deployed as part of the broader Datadog monitoring ecosystem to provide full stack visibility from agent reasoning to end user impact.

Whether you want to cut software costs or escape vendor lock-in, these open source alternatives to Datadog LLM Observability give you an option you fully own. You can self-host them, so your data stays on infrastructure you control. This page lists 1 open source alternative to Datadog LLM Observability. The most popular is OpenLLMetry. It uses the Apache-2.0 license.