tools
DeepEval
Open-source framework for testing and evaluating LLM apps, agents, RAG pipelines, chatbots, and multimodal systems.
DeepEval runs LLM evaluations locally through Python or TypeScript, with Pytest-style assertions, 50+ metrics, tracing, synthetic dataset generation, conversation simulation, and component-level testing. It supports agents, RAG pipelines, chatbots, MCP systems, and custom workflows.
Developers use the CLI in local development and CI/CD, including with coding agents such as Cursor, Claude Code, and Codex. Shared dashboards, regression tracking, observability, production monitoring, and other hosted features are provided through the separate Confident AI platform and may cost extra.
Features
- Run LLM evaluations locally with Pytest-style assertions
- Use 50+ metrics for agents, RAG, safety, conversations, and multimodal apps
- Trace agent runs and evaluate complete trajectories or individual components
- Generate synthetic datasets and edge-case goldens
- Run evaluation suites from a CLI in local development or CI/CD
- Support Python and TypeScript SDKs
- Integrate with LangChain, LangGraph, OpenAI Agents, and OpenTelemetry
- Apache 2.0 licensed open-source framework
Use cases
- Test chatbot answers for relevance, correctness, safety, and faithfulness
- Evaluate RAG retrieval and generated responses against reference data
- Grade agent trajectories, tool calls, and component-level behavior
- Run regression tests on every pull request in CI/CD
- Generate synthetic test datasets from documents or application code
- Inspect local traces while iterating with a coding agent