Dropbox Dash lets people ask questions across their scattered work tools and get a single answer back from a chat agent, rather than hunting through a dozen apps themselves. Getting that agent to actually help, rather than just sound confident, turned out to depend less on the model itself and more on how rigorously the team could measure what "helping" even means.
The team's starting point was a suite of LLM-as-judge evaluations. Rather than scoring only the agent's final reply, these judges inspect the entire path the agent took to get there: how it interpreted the user's intent, what context it pulled in, which tools it reached for, how it handled ambiguous phrasing, whether its answer was actually grounded in the material it retrieved, and whether the task got completed at all. That full-trajectory view matters because a chat agent can produce a plausible-sounding answer while still having misread the question or skipped a step it should have taken.
Once that evaluation layer existed, the next problem was a familiar one in applied machine learning: evaluation by itself doesn't improve anything. It just tells you where the gaps are. To close the loop between measurement and improvement, the team turned to DSPy, an open-source framework designed to optimize AI systems using feedback signals rather than hand-tuned prompts.
The approach ran in two stages. First, DSPy was used to calibrate the judges themselves against a small set of human-labeled examples, so the automated scores tracked more closely with what a person would actually judge as a good or bad response. This mattered because judges built on smaller, cheaper models tend to be brittle about output formatting. In one internally reported case, a lower-cost model used as a relevance judge produced malformed JSON output on more than 40 percent of its responses before optimization, a failure mode that silently discards evaluation signal since a response that can't be parsed gets treated as fully incorrect. After DSPy optimization, malformed outputs on that judge dropped by more than 97 percent, alongside a substantial improvement in how closely its scores tracked human ratings.
With a more reliable judge in place, the second stage used that judge's output as the optimization target for the chat agent's own system prompt. This created a feedback loop: human labels sharpened the judges, the judges produced evaluation signal at a scale no human review team could match, and that signal was then used to refine how the agent itself was instructed to behave.
The payoff showed up in two places at once. Users saw noticeably fewer incomplete answers, meaning the agent was more consistently finishing the job it was asked to do rather than stopping short. At the same time, the team was able to trim token usage without giving up answer quality, which matters at Dash's scale since every extra token in a system prompt or retrieved context is a cost that compounds across millions of queries.
The broader lesson generalizes well beyond Dropbox's specific product. Teams building agents often treat evaluation as a one-time scorecard: run the eval suite, note the score, move on to the next feature. Framing evaluation as an optimizable signal instead, something that can be fed back into prompt or judge design through frameworks like DSPy, turns a static report card into an engine that keeps improving both halves of the system: the judge that measures quality, and the agent being judged.