Frontier Coding Agents Use Metaprogramming to Adapt to Unfamiliar Programming Languages

Problem
This work addresses the gap in evaluating large language model (LLM)-based coding agents in unfamiliar programming languages, a scenario often overlooked in existing benchmarks. Most contemporary evaluations focus on mainstream languages and libraries, which may not accurately reflect an agent’s adaptability to esoteric languages. The authors highlight that existing benchmarks, such as SWE-Bench Verified and Terminal-Bench 2.0, compress performance differences, masking the true capabilities of these agents. This paper is a preprint and has not undergone peer review.

Method
The authors evaluate six contemporary coding agents, including Claude Opus 4.6 and GPT-5.4 xhigh, across four esoteric programming languages using a sequential setup that involves file editing, local execution, and hidden-test grading. The core technical contribution lies in the identification of metaprogramming strategies employed by stronger agents, where they generate target-language code using Python programs instead of writing directly in the target languages like Brainfuck and Befunge-98. The study also investigates the impact of providing Opus-derived Python helper code to weaker agents, such as Sonnet 4.6 and GPT-5.4 mini, to enhance their performance. The authors analyze the effects of interpreter calls and output tokens on agent performance, revealing that these resources amplify existing strategies rather than create new ones.

Results
The results indicate that the strongest agents, Claude Opus 4.6 and GPT-5.4 xhigh, significantly outperform weaker agents when allowed to use metaprogramming strategies. For instance, when metaprogramming is forbidden, performance drops sharply, demonstrating its critical role in adapting to unfamiliar languages. The introduction of Opus-derived Python helper code leads to substantial performance improvements for Sonnet 4.6 and GPT-5.4 mini, while Haiku 4.5 shows minimal gains. The study quantifies these improvements, although specific numerical performance metrics against named baselines are not disclosed in the abstract.

Limitations
The authors acknowledge that their evaluation is limited to a specific set of esoteric languages and may not generalize to all programming languages. Additionally, the reliance on metaprogramming strategies raises questions about the true understanding of the target languages by the agents. The study does not explore the long-term adaptability of these agents or their performance in real-world coding scenarios, which could provide further insights into their practical utility.

Why it matters
This research has significant implications for the development of coding agents, particularly in enhancing their adaptability to diverse programming environments. By demonstrating that strong agents can leverage metaprogramming to construct and debug code in unfamiliar languages, the findings suggest new avenues for improving agent performance in real-world applications. This work contributes to the ongoing discourse on the capabilities of LLMs in programming tasks, as discussed in related literature, and is available on arXiv.