SIGA: Self-Evolving Coding-Agent Adapters for Scientific Simulation

Problem
The paper addresses the challenge of agent-tool interface grounding in scientific simulation, specifically the need for coding agents to adapt to specialized input languages of simulators. Current coding agents lack the necessary understanding of a simulator’s executable contract, which includes its vocabulary, structural constraints, validation rules, and termination conditions. This gap results in significant time investment for domain scientists to learn and configure simulators. The work is presented as a preprint, indicating it has not yet undergone peer review.

Method
The authors propose SIGA (Simulator-Interface Grounding Adapter), which enhances coding agents by providing the required executable contract through several mechanisms: retrieval, procedural memory, in-trajectory validation, and validation-enforced termination. The architecture leverages a combination of these components to facilitate the coding agent’s interaction with the simulator. The primary evaluation is conducted on the GEOS multiphysics simulator, where SIGA is tasked with generating a complete simulation deck. The training compute details are not disclosed, but the evaluation focuses on the agent’s performance in terms of speed and accuracy.

Results
SIGA demonstrates a significant performance improvement over baseline coding agents. It produces a complete GEOS deck in approximately five minutes, achieving a TreeSim score above 0.90, which is comparable to an extended-budget human expert who requires about three hours, resulting in a 36x wall-clock speedup. On a more challenging held-out dataset, SIGA’s grounding mechanism improves the TreeSim score from 0.720 to 0.789, representing a 10% relative gain over the baseline agent. Additionally, the standard deviation across seeds is reduced by 16x, indicating enhanced consistency. The self-evolution capability of SIGA allows it to rewrite adapter contents based on prior trajectories, yielding the highest mean performance on held-out GEOS tasks and matching or surpassing the best hand-designed configurations. The transferability of SIGA to other simulators, such as OpenFOAM and LAMMPS, reveals that the importance of grounding mechanisms varies by interface, with validation being crucial for structural completeness and memory/retrieval for domain correctness.

Limitations
The authors acknowledge that the performance of SIGA may vary depending on the complexity of the simulator and the specific domain knowledge required. They do not address potential limitations related to the scalability of the approach across a broader range of simulators or the generalizability of the self-evolution mechanism. Additionally, the reliance on prior trajectories for self-improvement may introduce biases based on the quality of past interactions.

Why it matters
The implications of this work are significant for the field of scientific computing, as it suggests that lightweight, self-improvable grounding layers can effectively transform general coding agents into practical tools for operating complex scientific software. This advancement could reduce the barrier to entry for domain scientists, allowing them to leverage powerful simulation tools without extensive training. The findings contribute to ongoing research in agent-based systems and their applications in scientific domains, as published in arXiv.