LangWatch — the open-core platform to test, evaluate, and monitor LLM agents across multi-turn, multi-step conversations, not just single outputs.
What is LangWatch?
LangWatch is a testing, evaluation, and observability platform built for complex AI agents—voice bots, coding assistants, and multi-agent systems—where failures hide in interactions rather than individual responses. It takes agent traces, scenarios, and evaluation criteria as input and produces test verdicts, performance scores, cost breakdowns, and searchable trace histories. The platform runs as a managed cloud service (SaaS) or self-hosted via Docker/Kubernetes, built by the team at LangWatch.
Key Features
- Voice & text simulations — Simulate realistic multi-turn user conversations, including voice with audio effects like phone quality, using a user simulator agent that pushes your agent turn by turn (run a scenario or write one in Claude Code).
- Scenario-based testing — Turn plain‑language requirements into automated test scenarios that run locally and in CI, with the same setup for every pull request.
- Judge Agent — A multi‑step evaluator that reads the full trace (tool calls, LLM responses, timing) and returns a verdict with reasoning, supporting LLM‑as‑judge, custom code, and pairwise evaluation.
- Red teaming — Adversarial simulations that probe for jailbreaks, policy breaks, and unsafe tool calls before your users find them.
- OpenTelemetry-native observability — Trace every agent step, token, and cost across any framework (LangGraph, LiveKit, Vapi, ElevenLabs, OpenAI Agents, etc.) with a Cmd+K search over millions of traces.
- Langy AI test generator — Transform a product manager’s goal (e.g., “test refund flow with frustrated EU customers”) into a full scenario plan, run it, score with JudgeAgent, and draft a PR—median PM-to-PR time ~14 minutes.
- Whitebox & blackbox integrations — Connect via in‑process hooks or through the API, working with any agent framework without rewriting your agent.
Who is it for?
- AI engineers & agent builders — Use simulations and evaluations to catch regressions in multi‑turn conversations before shipping, then observe live production traces to find new issues.
- Product managers — Write plain‑language goals and let Langy generate test plans, so they own the spec without writing code.
- DevOps & platform teams — Add scenario tests to CI/CD pipelines, set up cost and latency dashboards, and enforce governance via RBAC, SSO, and audit logs.
What can you do with LangWatch?
- Test a voice agent with realistic user simulations — Run a scenario that simulates a frustrated caller demanding a refund, then verify the agent apologizes, escalates, and offers a goodwill credit. (Try the user simulator)
- Compare two models side‑by‑side — Use pairwise evaluation to decide whether GPT or Claude is better at categorizing products, scoring each output on exact match and granularity.
- Trace and debug a failed tool call — Open a trace, inspect every tool call (e.g., a Stripe refund mock), see latency and token cost, and replay the conversation to replicate the issue.
How does LangWatch work?
- Connect your agent — Integrate via OpenTelemetry or one of the framework wrappers; the platform supports both in‑process (whitebox) and API‑only (blackbox) connections.
- Create scenarios — Write a plain‑language goal or directly define multi‑turn test scripts in Claude Code; Langy can auto‑generate scenarios from a short brief.
- Run & evaluate — Execute scenarios locally or in CI. The Judge Agent scores each conversation against your rubric, and results are displayed with reasoning.
- Observe in production — Continuous trace ingestion provides latency, cost, and topic clustering; anomalous traces can be turned into new simulation scenarios.
