From Agent Traces to Trust: Evidence Tracing and Execution Provenance in LLM Agents

Problem
The paper addresses the lack of mechanisms for verifying, debugging, and auditing the behavior of large language model (LLM)-based agents, particularly as they interact with external tools and environments. While LLMs demonstrate increased autonomy in solving complex tasks, their outputs often lack transparency regarding the processes that led to them. This preprint highlights the need for a systematic approach to evidence tracing and execution provenance, which is currently underexplored in the literature.

Method
The authors propose a conceptual framework that organizes evidence tracing and execution provenance into a unified perspective. They introduce a taxonomy that categorizes trace sources, evidence and execution units, provenance relations, tracing granularity and timing, representation forms, and trust functions. Key methodological directions include:

• Provenance Representation: Developing structured formats for capturing the relationships between evidence and outputs.
• Evidence Attribution: Mechanisms to determine which pieces of evidence support specific claims.
• Tool-Use Provenance: Tracking the safety and justification of tool calls made by agents.
• Runtime Guardrails: Implementing safety mechanisms during agent execution.
• Provenance-Bearing Memory: Enhancing memory systems to retain provenance information.
• Trace-Based Observability: Creating methods for real-time monitoring of agent behavior.
• Failure Diagnosis: Establishing protocols for identifying and analyzing execution failures.

The authors also map existing benchmarks, datasets, and evaluation metrics to these provenance-related capabilities, advocating for a shift from evaluating final-answer correctness to assessing process-level accountability.

Results
While the paper does not present empirical results or quantitative comparisons against specific baselines, it emphasizes the importance of developing benchmarks that can evaluate the proposed provenance capabilities. The authors suggest that future work should focus on creating realistic execution-trace benchmarks and recovery-oriented evaluation metrics to assess the effectiveness of their framework.

Limitations
The authors acknowledge several limitations, including the need for unified trace schemas and the challenge of developing claim-level and semantic provenance. They also note the absence of privacy-aware audit infrastructure as a significant hurdle. Additionally, the paper does not provide empirical validation of the proposed framework, which may limit its immediate applicability.

Why it matters
This work is crucial for advancing the accountability and transparency of LLM agents, particularly as they become more integrated into critical applications. By establishing a framework for evidence tracing and execution provenance, the authors lay the groundwork for future research that can enhance the reliability of LLMs in real-world scenarios. The implications of this research extend to improving debugging processes, ensuring tool-use safety, and fostering trust in AI systems, as published in arXiv cs.AI.