Ex Ante Evaluation of AI-Induced Idea Diversity Collapse

Problem
This paper addresses a significant gap in the evaluation of creative AI systems, specifically the oversight of population-level diversity in generated outputs. Traditional assessments focus on individual utility, neglecting the phenomenon where increased similarity among outputs leads to diminished value—a concept termed “idea diversity collapse.” The authors propose a novel framework for benchmarking this collapse without relying on human-AI interaction data, which is particularly relevant given the increasing deployment of AI in creative domains. This work is presented as a preprint and has not yet undergone peer review.

Method
The authors introduce a human-relative framework that quantifies the risk of crowding in AI-generated ideas by modeling ideas as congestible resources. They derive two key metrics: the excess-crowding coefficient (Δ) and the human-relative diversity ratio (ρ). The condition (ρ \ge 1) indicates no excess crowding, while (Δ) is linked to an adoption game that incorporates exposure-dependent redundancy costs. The methodology involves generating outputs from three state-of-the-art large language models (LLMs) across various tasks—short stories, marketing slogans, and alternative uses—and comparing these outputs to matched human-generated baselines. The evaluation protocol is designed to stabilize estimates with feasible model-only sample sizes, allowing for effective assessment of crowding risk.

Results
The findings reveal that the three evaluated LLMs consistently fall below the parity condition (ρ \ge 1) across different crowding kernels, indicating significant excess crowding in their outputs. The authors provide quantitative results demonstrating that the models generate outputs that are less diverse than those produced by humans, leading to a higher likelihood of idea diversity collapse. The paper emphasizes that by adjusting generation protocols, it is possible to mitigate crowding, thus making diversity collapse a manageable target during the development phase of creative AI systems.

Limitations
The authors acknowledge that their framework does not account for the subjective quality of generated ideas, which may vary independently of diversity. Additionally, the reliance on model-only generations may overlook nuances present in human creativity that could influence crowding dynamics. The study is also limited by the specific tasks chosen for evaluation, which may not generalize across all creative domains. Furthermore, the framework’s effectiveness in real-world applications remains to be validated.

Why it matters
This research has significant implications for the design and evaluation of creative AI systems. By highlighting the importance of population-level diversity, it encourages developers to consider not just the quality of individual outputs but also their collective impact. The actionable insights regarding generation protocol adjustments provide a pathway for mitigating diversity collapse, fostering the development of AI systems that can produce a richer variety of ideas. This work lays the groundwork for future research into population-aware metrics and evaluation frameworks in creative AI.

Authors: Nafis Saami Azad, Raiyan Abdul Baten
Source: arXiv:2605.06540
URL: https://arxiv.org/abs/2605.06540v1