BazaarBench: Delegation Safety in Decentralized C2C Marketplaces Run by LLM Agents

Problem

The paper addresses a significant gap in the evaluation of safety mechanisms for large language model (LLM) agents operating in decentralized consumer-to-consumer (C2C) marketplaces. The authors highlight the lack of systematic approaches to assess how these agents manage transactions and ensure safety, particularly under various operational conditions. This work is presented as a preprint and has not undergone peer review.

Method

The authors propose BazaarBench, a simulated C2C marketplace designed to evaluate the performance and safety of LLM agents. Key components of the methodology include:

• Simulator: BazaarBench simulates a C2C marketplace environment.
• Transaction Tracking: The simulator tracks ownership, item condition, and commitments across transactions to ensure accountability.
• Evaluation Mechanism: The evaluation combines record checks with rubric-based judgments made by LLMs to assess transaction outcomes.
• Failure Types: The authors identify six distinct failure types that can occur across five stages of the transaction process.
• Market Setup: The simulation runs three base markets for 30 simulated days, involving 100 agents.
• Model Source: Inventories for the marketplace are drawn from a public eBay sample, providing realistic item listings.
• Continuation Runs: The study includes 45 continuation runs, each lasting seven simulated days, starting from the state at day 30.
• Agent Control: The model controls 20 selected agents, while the remaining 80 agents operate using a base model.
• Instructions: The models are tested under various conditions, including ordinary instructions, deadline pressure, and adversarial instructions to evaluate their robustness.

Results

The results indicate significant differences in performance based on the type of instructions given to the agents:

• Adversarial Completion Rate: The completion rate for committed transactions under adversarial conditions ranges from 15.4% to 33.4%, compared to ordinary instructions.
• GPT-5.4 Completion Rate: The GPT-5.4 model achieves a completion rate of 55.5% for committed transactions despite issues, again in comparison to ordinary instructions.
• Simulated Weekly Earnings: Average simulated weekly earnings increase from USD 20 under ordinary instructions to USD 33 under adversarial instructions, indicating a notable impact of instruction type on economic outcomes.
• Source of Earnings Increase: The increase in earnings is primarily attributed to transactions involving items that sellers never physically held, highlighting potential risks in the marketplace dynamics.

Limitations

The authors do not report any limitations in their study. However, the absence of reported limitations may suggest a need for further exploration of potential biases or constraints in the simulation environment.

Why it matters

This work has significant implications for the design and deployment of LLM agents in decentralized marketplaces. By providing a structured evaluation framework, BazaarBench can help researchers and practitioners identify safety vulnerabilities and improve the reliability of LLM agents in real-world applications. The findings also underscore the importance of instruction design in influencing agent behavior and transaction outcomes, which could inform future developments in agent-based systems.