Why Sampling Is Not Choosing: Intentionality, Agency, and Moral Responsibility in Large Language Models

Problem — This paper addresses the philosophical and ethical implications of attributing agency and moral responsibility to large language models (LLMs). It critiques the prevailing narrative in AI literature that suggests LLMs possess intrinsic intentionality and can be considered moral agents. The work is a preprint and unreviewed, contributing to ongoing debates about the nature of agency in AI systems.

Method — The author, Joseph Keshet, employs a philosophical analysis to dissect the concepts of agency, intentionality, and moral responsibility as they pertain to LLMs. The paper argues that moral responsibility necessitates commitment-bearing agency, which is fundamentally grounded in intrinsic intentionality and self-attributed actions. Keshet distinguishes between the probabilistic nature of LLM outputs and genuine choice, asserting that the variability introduced by stochastic sampling does not equate to authorship or moral agency. The analysis includes counterarguments to common philosophical positions such as the intentional stance, functionalism, and compatibilism, demonstrating that these frameworks fail to substantiate claims of agency in LLMs.

Results — The paper does not present empirical results or quantitative benchmarks, as it is primarily a philosophical discourse rather than an experimental study. Instead, it systematically dismantles the arguments supporting the notion of LLMs as moral agents, emphasizing that their outputs are the result of learned probabilistic mappings rather than intentional choices. The author’s conclusions are drawn from logical reasoning rather than experimental validation, making the work more theoretical in nature.

Limitations — The primary limitation noted by the author is the lack of empirical evidence to support claims of agency in LLMs, as the paper is grounded in philosophical argumentation rather than experimental data. Additionally, the paper does not explore potential frameworks for understanding agency in AI beyond the critiques presented, which may limit its applicability in practical AI ethics discussions. The author also does not address the implications of LLMs in real-world applications where perceived agency might influence user interactions or societal norms.

Why it matters — This paper is significant for researchers and engineers working on AI ethics, as it challenges the assumptions underlying the design and deployment of LLMs. By clarifying the distinction between probabilistic output generation and genuine agency, it encourages a more nuanced understanding of the moral implications of AI systems. This work is crucial for informing guidelines and policies regarding the use of LLMs in sensitive applications, as it underscores the importance of recognizing the limitations of these technologies in terms of moral responsibility. The insights presented are relevant for ongoing discussions in the field, as published in arXiv.