Evaluations as the foundation
Dropbox's Dash chat agent is judged on a suite of LLM-as-judge evaluations that look at the entire trajectory an agent takes to satisfy a user, not just the final response. The trajectory includes how the agent interprets intent, gathers context, uses tools, handles ambiguity, grounds its answer, and completes the task. Five dimensions matter: user intent following, semantic relevance, tool calling, instruction following, and context selection.
The evaluations are the powerhouses behind the judges. A judge for chat outcomes, given the context available to the agent, measures relevance, reasoning quality, evidence use, robustness, task completion, and alignment with the user's ask. Before any of this could improve the agent itself, the judges had to be trustworthy. The path to trustworthy judges is human-labeled examples and a consistent evaluation rubric.
The calibration step
Human evaluators reviewed a sampled set of internal chats, including the final responses and the trace logs showing how the agent arrived at them. Each example was scored across the five dimensions, on a 1 to 5 scale. The evaluators followed a structured process: first, did the agent understand the user's intent and select the right context; then, did the searches, retrievals, and tool actions gather the right information; then, were the claims in the final response supported by the selected evidence; finally, the response was scored for relevance, grounding, completeness, and instruction following.
The set of labeled examples is small. It has to be, because labeling is expensive. The size of the set is a constraint, not a target.
DSPy does the optimization
With human labels in hand, the team used DSPy to improve the judges themselves, calibrating them against the labeled set so their scores matched human judgment more closely. Then they used those improved judges to optimize the chat agent's system prompt. The algorithms in play are GEPA and MIPROv2, both of which take evaluation feedback and automatically propose prompt changes that improve against the labeled signal.
GEPA, in particular, works by replaying a candidate prompt on representative historical chats, scoring the resulting agent outputs with the evaluation pipeline, and using those scores as the feedback signal for the next prompt update. Each round of the loop produces a measurable improvement, and the improvements are real because they're validated against a human-aligned judge rather than a self-referential metric.
The result
The feedback loop is: human labels improve the judges, the judges produce scalable evaluation signals, and those signals improve the agent. Users saw significantly fewer incomplete answers. Token usage went down. Answer quality did not compromise. The system got better on a metric the team trusted, because the metric was built on a foundation of human labels and the optimization was constrained by that foundation.
The interesting design choice is what DSPy was not asked to do. It was not asked to write the chat agent's behavior from scratch. It was not asked to invent new product features. It was asked to optimize a prompt against a well-defined evaluation, where the evaluation itself had been calibrated to human judgment. The scope of the optimization was the prompt, and the scope of the prompt was the agent's behavior. That is a much smaller surface than "let the AI figure out the chat agent," and it is the reason the optimization worked.