LP-Eval: Rubric and Dataset for Measuring the Quality of Legal Proposition Generation

Problem
This paper addresses the gap in the automatic generation and evaluation of legal propositions within the domain of Legal Natural Language Processing (NLP). Despite the importance of legal proposition generation for legal reasoning and doctrinal scholarship, it remains under-explored in the literature. The authors present LP-Eval, a novel evaluation framework co-developed with legal experts, to systematically assess the quality of legal propositions generated by large language models (LLMs). This work is presented as a preprint and has not yet undergone peer review.

Method
The core technical contribution is the LP-Eval framework, which consists of a three-step evaluation rubric that dissects legal proposition quality into formal validity and substantive dimensions. The authors utilize a dataset comprising 100 legal propositions generated by LLMs, annotated by two legal experts. The evaluation rubric is designed to capture both the structural integrity and the substantive relevance of the propositions. The authors also explore the use of LLMs as evaluators, comparing their rubric-guided judgments against expert assessments. The training compute and specific architectures of the LLMs used for generation are not disclosed in the paper.

Results
The findings indicate that LLMs can produce predominantly well-formed legal propositions, achieving a high-quality score as per the LP-Eval rubric. Notably, propositions derived from well-established legal cases received higher quality ratings compared to those generated from more recent cases. The authors report that LLMs, when guided by the rubric, yield evaluations that align more closely with expert assessments than when providing direct overall scores. However, the LLMs exhibit insensitivity to nuanced distinctions that human experts can discern, suggesting limitations in their evaluative capabilities.

Limitations
The authors acknowledge several limitations, including the reliance on a relatively small dataset of 100 propositions, which may not be representative of the broader legal landscape. Additionally, while the rubric improves alignment with expert evaluations, it does not fully capture the complexity of legal reasoning, particularly in finer-grained distinctions. The paper does not address potential biases in the LLMs or the implications of using LLMs as evaluators in legal contexts, which could affect the reliability of the assessments.

Why it matters
This work has significant implications for the field of Legal NLP, as it provides a structured approach to evaluating the quality of legal propositions generated by LLMs. The introduction of the LP-Eval rubric could serve as a benchmark for future research, facilitating the development of more sophisticated legal reasoning systems. Furthermore, the findings highlight the potential of LLMs in legal applications while also underscoring the need for careful evaluation and the integration of human expertise in legal decision-making processes.

Authors: Shanshan Xu, Johan Lindholm, Amogh Raina, Henrik Palmer Olsen, Daniel Hershcovich
Source: arXiv:2605.19815
URL: https://arxiv.org/abs/2605.19815v1