The explosion of AI agent marketplaces has created an unexpected problem: abundance paradox. With thousands of agents now available across platforms like OpenAI's ecosystem, Anthropic's Claude ecosystem, and various open-source repositories, the challenge of agent discovery has shifted from scarcity to signal degradation. Unlike traditional API search, where function signatures and documentation provide clear specification contracts, agent capabilities are fundamentally compositional and execution-dependent—their true utility emerges only through runtime interaction with specific task contexts. This gap between advertised functionality and actual performance in the wild remains largely unexplored, until now.

Researchers have introduced AgentSearchBench, a large-scale evaluation framework that formalizes agent discovery as a retrieval and reranking problem grounded in execution outcomes. Built on a corpus of nearly 10,000 agents harvested from multiple providers, this benchmark represents the first systematic study of agent search under realistic conditions. Rather than assuming well-curated, narrowly-defined agent pools with clean documentation—the typical assumption in prior work—AgentSearchBench embraces the messiness of real-world agent ecosystems where descriptions are inconsistent, capabilities overlap ambiguously, and true performance emerges only through execution.

The benchmark formalizes two complementary search scenarios. The first evaluates retrieval and reranking under executable task queries—where the evaluation harness can actually run candidate agents against ground-truth tasks and measure success. The second, more challenging setting operates under high-level task descriptions alone, forcing rankers to operate without execution feedback. This distinction matters because it isolates two different failure modes: semantic retrieval failures (finding agents whose descriptions sound relevant but perform poorly) and ranking failures (ordering candidates appropriately given only textual signals). Relevance judgments are grounded in actual execution performance—not human annotations—making this an unusually rigorous evaluation methodology for the agent search domain.

The empirical findings are striking and consequential: semantic similarity metrics show consistent and substantial gaps with actual agent performance. Dense retrieval methods using standard embedding models frequently rank semantically similar agents high while their execution-time success rates remain mediocre. This finding challenges a core assumption in information retrieval and semantic search: that cosine similarity in embedding space correlates with task-specific utility. For agents, the relationship is far more complex. An agent's description may emphasize certain capabilities while its internal architecture, error handling, and compositional structure determine real-world reliability in ways that textual descriptions simply cannot capture.

More intriguingly, the researchers demonstrate that lightweight execution-aware probing signals—behavioral indicators derived from test-time interaction with agents—substantially improve ranking quality. Rather than executing agents on full task suites (computationally expensive), these behavioral signals capture execution patterns through minimal probing: Does the agent handle edge cases gracefully? How does it respond to ambiguous inputs? What is its latency profile? These signals, though cheaper than full execution evaluation, contain critical information about agent robustness that semantic methods miss entirely. This suggests a promising middle ground between pure semantic ranking (fast but inaccurate) and exhaustive execution evaluation (accurate but expensive).

Within the broader AI systems landscape, AgentSearchBench addresses a critical infrastructure gap. As agent ecosystems mature and agent composition becomes standard practice—where complex workflows chain multiple agents together—the cost of poor agent selection compounds. A suboptimal choice early in a composition pipeline can cascade into failures downstream. This benchmark provides both a rigorous evaluation framework and an empirical foundation for building better agent discovery systems. The work also raises architectural questions: Should agent platforms embed execution-aware metadata into agent registries? Should agent discovery APIs return not just relevance scores but also confidence intervals derived from behavioral signals?

CuraFeed Take: This benchmark exposes a fundamental mismatch between how we currently think about agent discoverability and how agents actually work in practice. The finding that semantic similarity fails to predict execution performance is not surprising in hindsight—agents are stateful, compositional systems with emergent behaviors—but it's critical that this is now empirically quantified at scale. The practical implication is clear: agent platforms and frameworks that ignore execution signals in their discovery mechanisms are leaving substantial performance on the table. We should expect to see agent registries evolving toward hybrid ranking systems that combine semantic retrieval with lightweight behavioral probing, similar to how modern recommendation systems blend collaborative filtering with user interaction signals. The researchers' emphasis on execution-grounded evaluation also sets a methodological precedent: future agent benchmarks should prioritize actual task performance over human annotation, raising the bar for what constitutes rigorous agent evaluation. Watch for agent platform providers to invest in execution-aware indexing and ranking infrastructure—this is where competitive advantage will increasingly concentrate.