The Problem With Judging an Agent
Evaluating a chatbot used to mean checking whether the final answer was right. Evaluating an agent is a different kind of problem entirely. Dropbox's Dash chat agent doesn't just answer questions, it interprets intent, decides which tools to call, retrieves from documents and messages, and sometimes has to ask for clarification across multiple turns. Judging that whole trajectory, not just the last sentence it produces, is what Dropbox's machine learning team set out to do, and then use as fuel for actually improving the agent.
Teaching the Judges Before Trusting Them
Before any optimization could happen, Dropbox needed judges it could trust. The team sampled real internal chats along with the trace logs showing how the agent reached its answers, then had human evaluators score each one across five dimensions: intent following, semantic relevance, tool calling, instruction following, and context selection.
Evaluators didn't just assign a number. They wrote reasoning notes and tagged specific failure types, things like stale evidence or unsupported claims, which gave the team a much richer signal than a bare score ever could. That labeled dataset became the benchmark against which every future automated judge would be measured.
DSPy Enters the Picture
With labeled data in hand, Dropbox turned to DSPy, an open-source framework for optimizing AI systems using evaluation feedback, applying algorithms like GEPA and MIPROv2 to rewrite and refine the judges' own instructions until their scores lined up with human judgment. In some cases DSPy rewrote a judge's rubric from scratch; in others it adapted an existing judge to a new underlying model, or targeted a single failure mode like a judge over-scoring outdated information.
Once the judges were reliable, Dropbox pointed the same machinery at the chat agent's system prompt itself. Each candidate prompt generated by GEPA was replayed against representative historical chats, scored by the calibrated judges, and the resulting feedback shaped the next round of proposals, replacing what used to be a slow, manual cycle of engineers reading failures and hand-editing prompts.
The Payoff
The numbers were hard to argue with. In the first two weeks of using the automated loop, the team generated six prompt candidates, nearly double the five manual changes made in the entire prior month. After launch, incomplete answers dropped by 26% and missed key aspects of user requests fell by 13%, with the improvements showing up within a day. Token usage also fell by 5.4% and average response length shrank by 9.8%, all without sacrificing answer quality.
A Guardrail-First Approach to Automation
Dropbox is careful to note that automated prompt rewriting still needs limits. Most edits were constrained to small, targeted instruction changes, with automated checks for prompt structure, size limits, and caching behavior built in to keep candidates production-safe.
The broader lesson is less about DSPy specifically and more about treating prompts the way machine learning teams already treat models: as measurable, optimizable artifacts, not static text someone writes once and forgets. As more companies run agents in production, expect this evaluation-driven feedback loop, human labels calibrating judges, judges scoring candidates, candidates competing against a stable baseline, to become the default way agent prompts get built rather than the exception.