“Up” is not the same as “working”
We called a real tool on 55,592 servers. Only 29,527 answered correctly. A server can be online, spec-compliant and still useless to your agent.
A health check tells you the door opens. It does not tell you anyone is home. So we stopped asking whether MCP servers respond and started asking whether they actually do the thing they advertise.
We called their tools
A smoke test picks one real tool from a server, sends it a minimal valid call, and checks that the answer comes back in the shape the schema promised. Across the catalogue that is 55,592 attempts so far. Here is the result:
smoke tests run 55,592
smoke tests passed 29,527 (53%)Just over half. And these are servers that answered the handshake, listed their tools and, in most cases, reported a healthy uptime percentage all week.
Where it breaks
The failures are rarely dramatic. A tool is listed but not implemented. A required parameter is not in the schema. The call succeeds and returns an error string inside a success envelope. An endpoint needs a tenant id nobody documented. None of this shows up in uptime.
What to take from it
- Uptime answers “is it reachable”, not “does it work”.
- A tool list is a claim, not a guarantee — the only proof is calling one.
- If you depend on a server, call its tool on a schedule, not just its health endpoint.