Version- and Scope-Aware Question Answering over Normative Documents: A Deployed System and an End-to-End Evaluation at Production Scale

Problem

This work addresses the challenge of answering questions based on normative documents while considering various contextual factors such as version, jurisdiction, subject, date, and source text traceability. The authors highlight the need for a robust system that can effectively manage these complexities in a production environment. Notably, this paper is a preprint and has not undergone peer review.

Method

The authors utilized a dataset comprising approximately 73,000 candidate normative documents to train and evaluate their system. The evaluation method involved a stratified sample of 200 questions drawn from a published benchmark, with each question linked to a gold source document. The system was compared against a hosted service, with the scoring methodology based on unrounded means. The governed system achieved an overall score of 97.7, while the hosted service scored 88.1, resulting in a significant gap of 9.6 points. The authors have made public resources available, including the question set, answer text, scores, and scripts for reproducing benchmark statistics.

Results

The governed system scored 97.7 on the evaluation benchmark, significantly outperforming the hosted service, which scored 88.1. This demonstrates the effectiveness of the proposed system in a real-world application.

Limitations

The authors did not report any limitations in their work, and no obvious limitations are noted in the available text.

Why it matters

The implications of this research are substantial for downstream applications in legal and regulatory domains, where accurate question answering over normative documents is critical. The high performance of the governed system suggests that it can be effectively deployed in production settings, potentially improving access to legal information and enhancing decision-making processes.