TRACE: A Metrologically-Grounded Engineering Framework for Trustworthy Agentic AI Systems in Operationally Critical Domains

Problem
This paper addresses the lack of a comprehensive engineering framework for developing trustworthy agentic AI systems in operationally critical domains. The authors identify a gap in existing literature regarding the integration of classical machine learning (ML) and large language models (LLMs) within a structured architecture that ensures accountability and reliability. The work is presented as a preprint and has not yet undergone peer review.

Method
The core technical contribution of this paper is the TRACE framework, which consists of a four-layer reference architecture designed to facilitate the development of trustworthy AI systems. The architecture includes:

1. Layer 1 (L1): A foundational layer that encompasses the data and model inputs.
2. Layer 2 (L2): Divided into two sub-layers (L2a and L2b), where L2a focuses on classical ML approaches and L2b on LLMs, allowing for a deliberate choice of model type based on the application context.
3. Layer 3 (L3): Implements a stateful orchestration-and-escalation policy that governs the interaction between AI systems and human operators.
4. Layer 4 (L4): Incorporates bounded human supervision to ensure oversight and accountability.

Additionally, TRACE introduces a metrologically grounded trust-metric suite aligned with standards such as GUM, VIM, and ISO 17025, which provides a framework for quantifying trustworthiness. The Computational Parsimony Ratio (CPR) is proposed as a design principle to quantify model parsimony, ensuring that the complexity of the AI system is justified by its performance.

Results
The authors demonstrate the applicability of TRACE through three case studies: clinical decision support, industrial multi-domain operations, and a judicial AI assistant. While specific quantitative results are not provided in the abstract, the framework’s design is intended to facilitate performance evaluation across these diverse domains. The effectiveness of the CPR in promoting model parsimony is highlighted, although comparative performance metrics against established baselines are not explicitly detailed in the provided text.

Limitations
The authors acknowledge that the TRACE framework is still in its conceptual phase and requires empirical validation across various operational contexts. They do not address potential challenges related to the scalability of the framework or the integration of diverse data sources. Additionally, the reliance on human supervision may introduce variability in system performance, which is not discussed in detail.

Why it matters
The TRACE framework has significant implications for the development of trustworthy AI systems in critical applications, where reliability and accountability are paramount. By establishing a structured approach that integrates classical ML and LLMs, TRACE encourages the adoption of best practices in AI engineering. The introduction of the CPR as a design principle may influence future research on model complexity and performance trade-offs, promoting a more rigorous evaluation of AI systems in operationally critical domains.

Authors: Serhii Zabolotnii
Source: arXiv:2605.03838
URL: https://arxiv.org/abs/2605.03838v1