Shipping a skill that calls an API is the easy part. Knowing whether it's actually working, and whether a recent change quietly broke something, requires running it against a set of test scenarios and scoring the results, repeatedly, across iterations. Skip that step and you're left testing manually and hoping for the best, which is exactly what most skill builders end up doing, for reasons that have less to do with laziness than with a set of genuinely awkward tradeoffs.
The first problem is cost. If the API belongs to an external service, every evaluation run is a real bill. Running 50 scenarios across three models with five repetitions each adds up to 750 API calls in a single session, and that number multiplies every time a prompt gets tuned or a different model gets tried. Even when the API is free to call, a second problem shows up as soon as the skill performs writes: PATCH calls mutate state, DELETE calls remove records, and an eval harness making those calls is changing live data as a side effect of simply measuring quality. Without a manual reset between runs, later evaluation results end up contaminated by state left over from earlier ones.
A subtler third problem appears when other people or systems are also writing to the same API. A score that drops between two otherwise identical eval runs might not reflect anything the skill builder changed; it might just be a colleague on another team updating a record the test scenarios happened to depend on. That's the same category of problem that plagues any evaluation process prone to non-determinism, just expressed through shared API state rather than model temperature or file ordering.
Faced with those tradeoffs, most teams skip rigorous evaluation entirely. They test manually, ship, and move on, which produces skills that work fine on a Tuesday demo and quietly break on Friday once the underlying API returns slightly different data, with nobody the wiser because nobody was measuring continuously.
The conventional fix is a mock server: a local stand-in that mimics the real API's responses, letting evals run in isolation against localhost. In practice that means writing and maintaining a fake server that handles multiple endpoints, supports filtering and pagination, returns realistic payloads, and correctly implements PATCH and DELETE semantics. For a small API that's an afternoon's work; for anything more substantial, it's a project that starts drifting out of sync with the real API the moment someone ships a change on the other side.
There's a second, more fundamental problem with mocks, though: pointing a skill at localhost means changing the URLs inside the skill file itself, which means the skill being evaluated is no longer the skill that will actually ship. Every token in a model's context shapes its reasoning, and swapping a production URL for a localhost address is a different set of tokens feeding that reasoning, which quietly introduces a variable into the very thing being measured.
The better-calibrated bar turns out to be lower than most people assume. A skill doesn't need a full replica of the backend, just stable URLs and realistic payloads with correct HTTP semantics for whichever operations it actually performs. Tools built around this idea, such as Dev Proxy, achieve it through a JSON configuration file that defines a base URL, a set of endpoints, and seed data, then intercepts HTTP traffic from the agent and answers locally without any server to build or maintain. The skill keeps calling the same real-looking URLs it always would, with no awareness that a proxy rather than the production server is answering.
Because the seed data resets on every run, there are no production records at risk, no ongoing API costs, and no awkward conversation about who deleted the test orders. Removing the proxy afterward means the skill goes straight back to production with zero code changes required, since the URLs never moved in the first place.
The underlying principle isn't really specific to AI agents at all: nobody runs integration tests against a production database, and there's no good reason to run eval scenarios against a production API either. Mock servers work, but they demand upkeep. Proxy-based emulation is lighter and keeps the skill under test identical to the one that ships. Either approach beats the alternative, which is shipping skills nobody has ever systematically measured and finding out they're broken only after a user does.