Your Hypothesis Is Probably Wrong
You've built an eval, found the gaps, and now you have a theory: rewrite the docs, tweak the CLI output, add a warning. It sounds reasonable. Ship it, measure again, right? Don't, not yet. LLMs don't reason the way we expect, and changes that seem obvious produce results that are anything but.
The Tip Nobody Followed
Working on a SharePoint Framework project upgrade scenario, a team needed an agent to discover a CLI tool from documentation. The docs included a tip recommending exactly that tool. The agent ignored it in every single run, despite reading the page every time.
The explanation turned out to be simple: the agent had already formed a plan from its training data before it even fetched the page. The documentation confirmed pieces of that plan, package names, version numbers, and the tip got filed away as optional noise rather than a directive. A dozen variations, repositioning the tip, rewording it, even including the exact command, changed nothing.
What finally worked wasn't a better tip. It was a warning that told the agent its current plan would fail, not that an alternative existed. Five out of five runs switched to the CLI. The agent abandoned its own plan because it was told the plan was broken, not because something better was dangled in front of it.
A Dozen Hypotheses, Most of Them Wrong
The surprises kept coming. Removing links to a competing migration guide boosted CLI adoption but tanked overall quality, since the runs that still missed the CLI no longer had the guide to fall back on. Structured JSON output improved configuration accuracy, but switching to JSONL, which looks nearly identical, cost far more tokens for worse results across the board. PowerShell syntax in code examples bled into unrelated parts of the agent's work. And handing the agent a ready-to-run script didn't mean it would run it; in every test, the agent decomposed the script and applied its steps individually instead.
Each of these contradicts a reasonable assumption. That's the point.
Why Emulation Beats Shipping and Praying
Changing a live documentation page means a pull request, a review, a deploy, then a measurement, and if the hypothesis was wrong, another PR to undo it. Changing an MCP server's response format means shipping a new version to test something that might not even work. That loop runs in days. Intercepting requests and returning modified responses locally, as if the change were already live, runs in minutes. The agent calls the same URLs and has no idea a proxy answered instead of the real server.
That difference is the whole game. Testing three ideas a week versus thirty isn't a minor efficiency gain, it changes what's actually knowable about how an agent behaves before something ships to real users.
The Real Skill Is Measuring, Not Predicting
Nobody on this team predicted the tip would fail or that the script wouldn't get executed. They were wrong on most of their assumptions, and that's fine, because the cost of finding out was cheap. If your agent reads documentation, you can emulate documentation changes. If it calls an API, you can emulate the response. Pick whatever surface it touches most, form a hypothesis, and test it before you ship anything at all.