It's tempting, once an eval has surfaced a gap in how an AI agent behaves, to jump straight to fixing it: rewrite the documentation, tweak the CLI output, ship the change, and measure again. A team working on agent experience found that instinct leads to a lot of wasted pull requests, because language models don't reason about instructions the way a person assumes they will, and seemingly obvious fixes can produce results nobody predicted.
The clearest example came from a SharePoint Framework project upgrade scenario, where an agent needed to discover a CLI tool through documentation. The docs included a tip recommending that tool. The agent read the page in every test run and ignored the tip every single time. The team's working theory was that a recommendation embedded in the page would nudge the agent's behavior; the reality was that the agent had already formed a plan from its training data before it even fetched the page, and the documentation only confirmed pieces of that existing plan, like package names and version numbers. The tip got filed away as optional context rather than treated as a directive, because that's functionally what it was: advisory language doesn't override a plan the model already believes will work.
The team tried more than a dozen variations to fix this. They repositioned the tip on the page, reworded it more forcefully, and even included the exact command to run. None of it moved the needle, which pointed to a more useful mental model: agents scan an entire page and gravitate toward actionable, step-by-step content, and a tip, by its very nature, isn't that. What actually worked was reframing the message as a warning that the agent's current plan would fail outright, rather than an alternative worth considering. Five out of five test runs flipped to using the CLI once the documentation told the agent its approach was broken, not because a better option existed, but because its existing plan had been invalidated.
That single finding was one of more than a dozen hypotheses tested during the same engagement, and the overall hit rate wasn't encouraging. Removing links to a competing migration guide improved CLI adoption but tanked overall quality, since the majority of runs that still failed to find the CLI no longer had the competing guide to fall back on either. Adding structured JSON output to the CLI improved configuration correctness meaningfully, but switching to the superficially similar JSONL format cost more than two and a half times the tokens with worse results across every measured dimension. PowerShell syntax embedded in code fences bled into the agent's other work, degrading code quality even in files that had nothing to do with the upgrade task. And providing an explicit, ready-to-run script did not cause the agent to execute it; across ten separate runs, the agent never once piped the script to a shell, instead decomposing it and applying the steps individually.
Each of these results contradicts an assumption that would have seemed entirely reasonable going in. The team's answer to that unpredictability wasn't to try to reason more carefully about LLM behavior in advance, but to make testing hypotheses cheap enough to run dozens of them rather than three. If a documentation change lives on a production site, testing a hypothesis means opening a pull request, waiting for review and deployment, and then measuring, with another PR required if the change turns out to be wrong or actively harmful. If it's an MCP server response format, testing means shipping a new version to real users before knowing whether the idea even works.
The alternative the team adopted was request interception: using a tool to intercept the agent's outbound requests and return modified responses as though a change were already live, without touching the actual production page or service. For the SharePoint work, that meant intercepting calls to Microsoft Learn and returning a version of the page with the proposed warning callout already added. When a hypothesis worked, the real change got shipped; when it didn't, like most of the tip-only experiments, the team moved on within minutes rather than days. The same interception pattern applies just as well to MCP servers, letting a team test a different response format without deploying anything new.
The deeper point is about experimental discipline rather than any single trick. Optimizing a conventional REST API is predictable: a faster query returns faster, a smaller payload costs less. Optimizing for how an LLM-driven agent behaves is not predictable in the same way, and a change that looks cosmetic can flip behavior entirely while a change that looks substantial does nothing at all. The only way to know which is which is to measure, and measuring only pays off if each experiment is cheap enough to run in volume. Teams that improve agent experience quickly aren't necessarily better at guessing how a model will react; they've just shortened the loop enough to test far more guesses per week.