LangChain explains how it evaluates its open-source Deep Agents framework using Harbor, an evaluation runner that requires specifying an agent, dataset, and sandbox environment. Rather than relying on simple LLM output checks, the approach favors end-to-end tests that run agents in real environments and validate the artifacts they produce. The team maintains three benchmarks covering autonomous work, multi-turn conversation, and retrieval tasks, running tasks multiple times to account for nondeterminism and keeping a faster “lite” benchmark for quick iteration.
