How GitHub Fixed Copilot Code Review: A Tale of Tools and Workflows
Imagine this: you give an AI agent better tools, expecting it to perform better. But instead of improving, things get worse. That's exactly what happened when GitHub tried to upgrade the tools powering Copilot's code review feature. The fix? It wasn't about the tools—it was about the instructions guiding how the tools were used.
The Problem: Better Tools, Worse Results
GitHub Copilot's code review feature is designed to analyze pull requests, explore surrounding code, and identify issues before they ship. Initially, it used its own set of code exploration tools. When GitHub decided to swap in better-maintained, shared tools from the Copilot CLI (like grep, glob, and view), they expected a seamless upgrade. But the results were disappointing: the cost of reviews increased, and fewer issues were caught.
The culprit? The instructions guiding the tools. The new tools were designed for broader coding tasks, not the specific workflow of code review. This mismatch led to inefficiencies, with the agent behaving like it was browsing a repository instead of focusing on the pull request.
The Fix: Rewriting Instructions for a Reviewer's Workflow
To solve the problem, GitHub rewrote the tool instructions to match how a human reviewer would approach a pull request. Instead of broad exploration, the new instructions emphasized a targeted approach:
- Start from the diff: Focus on the changes in the pull request.
- Ask specific questions: Narrow down the search to find relevant context.
- Batch searches before reading files: Avoid overloading the review with unrelated code.
- Read only when necessary: Use
viewto read exact evidence, not entire files.
This shift changed the agent's behavior from 'browse, read, search again' to 'ask, narrow, read, decide.' The result? A roughly 20% reduction in average review cost, with no drop in quality.
The Power of Workflows
This story underscores an important lesson: tools are only as good as the workflows that guide them. Shared tools can benefit multiple products, but only if the instructions and benchmarks align with the specific job at hand. In GitHub's case, the same tools (grep, glob, view) worked differently for code review and broader coding tasks because the workflows were tailored to each use case.
For developers building AI-powered systems, this means paying close attention to how tools are used, not just what tools are available. Small changes in instructions can lead to significant improvements in cost, quality, and efficiency.
The Takeaway
GitHub's experience with Copilot code review is a reminder that better tools don't always lead to better outcomes. The key is aligning tools with the right workflows and instructions. By doing so, GitHub not only improved Copilot's performance but also demonstrated the importance of tailoring AI systems to the tasks they're designed to solve.
So, the next time you're tempted to upgrade tools to fix a problem, remember: the solution might lie in rethinking how those tools are used.
Interested in more stories like this? Check out our biweekly newsletter for tips, technical guides, and best practices for developers.