When Skills Don't Help: A Negative Result on Procedural Knowledge for Tool-Grounded Agents in Offensive Cybersecurity

Problem
This preprint addresses a significant gap in the understanding of the utility of procedural knowledge, encapsulated as “Agent Skills,” in tool-grounded agents within the domain of offensive cybersecurity. While prior studies have indicated an average improvement in task pass rates with the introduction of Skills, this work highlights the inconsistency of these benefits across various tasks, with 16 out of 84 tasks exhibiting negative performance deltas. The authors aim to clarify the conditions under which Skills are beneficial versus redundant, particularly in a domain that has not been extensively evaluated against existing Skills benchmarks.

Method
The authors conducted a re-analysis of a previously published controlled study involving 180 runs of a multi-component (MCP)-grounded autonomous Capture-the-Flag (CTF) agent. They categorized the documentation conditions into four distinct ablation settings: No-Skills, Experiential-Skills, Curated-Skills, and Comprehensive-Skills, corresponding to increasing documentation richness (55, 1,478, 1,976, and 4,147 lines, respectively). The study employed statistical tests, including the Cochran-Armitage trend test and Cohen’s h effect size calculations, to evaluate the performance differences across these conditions. The authors propose that the critical variable influencing the effectiveness of Skills is the “environment-feedback bandwidth,” which refers to the quality and speed of feedback provided by the agent’s tool layer.

Results
The results indicate that the performance gap between the No-Skills and Comprehensive-Skills conditions is only 8.9 percentage points, with statistical significance not achieved (p = 0.71 for chi-squared; p = 0.25 for Cochran-Armitage). Furthermore, five out of six pairwise Cohen’s h values were below the 0.2 threshold, indicating small effect sizes. In specific scenarios, such as the timing side-channel setting, the introduction of Skills not only failed to improve performance but actively degraded it, suggesting that the procedural knowledge provided by Skills may be unnecessary or even counterproductive in certain contexts.

Limitations
The authors acknowledge that their findings are limited to the specific context of offensive cybersecurity and the particular tasks evaluated in the CTF setting. They do not explore the broader applicability of their conclusions to other domains or types of tasks. Additionally, the study’s reliance on a controlled experimental setup may not fully capture the complexities of real-world applications. The authors also note the need for further research to validate their hypothesis regarding environment-feedback bandwidth across different agent architectures and task domains.

Why it matters
This work has significant implications for the design and deployment of AI systems that utilize procedural knowledge. By elucidating the conditions under which Skills may be redundant, it challenges the prevailing assumption that more procedural knowledge always leads to better performance. The findings suggest that in environments with rich feedback mechanisms, the reliance on Skills could be minimized, leading to more efficient agent designs. The authors propose a falsifiable hypothesis that could guide future research and development in compound AI systems, emphasizing the importance of feedback quality in agent performance.

Authors: Samuel Jacob Chacko, James Hugglestone, Chashi Mahiul Islam, Xiuwen Liu
Source: arXiv:2605.20023
URL: https://arxiv.org/abs/2605.20023v1