Building a chat application with AI is easy. Building one that consistently provides helpful, accurate, and safe responses is hard. The gap between a working prototype and a production-ready system is filled with evaluations, feedback loops, and continuous improvement.
Netflix's Dash chat team faced this challenge head-on. They needed a way to systematically improve their AI responses based on real-world feedback. They found their answer in DSPy.
What is DSPy?
DSPy is a framework for programming—rather than prompting—language models. Instead of hand-crafting prompts and hoping they work, DSPy lets you define the task and the evaluation metrics, then automatically optimizes the prompts and the model's behavior.
It's a shift from prompt engineering to programmatic optimization. You tell DSPy what you want the model to do and how you'll measure success, and it figures out the best way to achieve it.
The Dash chat challenge
Dash is Netflix's internal productivity platform. It includes a chat feature where employees can ask questions about company policies, engineering practices, and more. The AI needed to provide accurate, helpful responses, but the team was struggling to maintain quality as the system scaled.
Manual prompt tuning was slow and brittle. Every change required extensive testing. And even when prompts worked, they often failed in edge cases.
The DSPy approach
The team used DSPy to automate the optimization process. They defined the task (answering questions about Netflix internal topics), the inputs (user questions), and the outputs (AI responses). They also defined evaluation metrics: accuracy, relevance, safety, and helpfulness.
DSPy then experimented with different prompts, model configurations, and reasoning strategies to find the combination that maximized the evaluation scores. The system learned from its own failures, improving over time.
Results
The results were significant. The team was able to improve response quality across all metrics, with fewer manual interventions. The system became more robust—it handled edge cases better and degraded more gracefully when things went wrong.
Equally important, the team gained confidence. They could make changes with the assurance that DSPy would help them maintain quality. The evaluation framework gave them visibility into what was working and what wasn't.
Lessons learned
Evaluations are the foundation. You can't improve what you can't measure. The team invested heavily in building a comprehensive evaluation suite, and it paid off.
Automation beats manual tuning. DSPy's ability to experiment at scale far exceeded what the team could do manually. The system found optimizations that humans wouldn't have considered.
Iterate continuously. The work didn't stop when the system went live. The team continued to collect feedback, update evaluations, and run DSPy optimizations, ensuring the system kept improving.
The future
The Dash chat team is now exploring other uses for DSPy, including personalization and multi-turn conversations. The framework has become an essential part of their AI development workflow.
For any team building AI-powered applications, DSPy offers a promising path to systematic improvement. It turns evaluation feedback into direct model improvements, closing the loop between what users experience and what the system learns.