RubricsTree: Scalable and Evolving Open-Ended Evaluation of Personal Health Agents across Health Memory and Medical Skills

Problem
The paper addresses the challenge of evaluating LLM-empowered personal health agents, which leverage user health metrics to improve healthcare access. Current evaluation methods are limited by the high cost and low scalability of physician annotations, while LLM-based evaluators suffer from subjectivity and inconsistency. This work is a preprint and unreviewed, highlighting the need for a robust evaluation framework that can scale effectively while maintaining clinical alignment.

Method
RubricsTree introduces a hierarchical taxonomy comprising over 100 atomic, clinically-verifiable Boolean rubrics, developed through an iterative human-in-the-loop curation process involving insights from 4,000 real user queries and an expert panel led by a physician. The framework employs a context-aware adaptive router that activates only the relevant subset of rubrics for each query, ensuring efficient evaluation. The auto-weighted rubric selection enhances the evaluation process by aligning it with expert standards. The authors conducted a systematic meta-evaluation to validate the framework’s effectiveness against a strong large-scale evaluation baseline.

Results
RubricsTree demonstrated significant improvements in expert alignment on challenging open-ended queries, outperforming the baseline evaluation method. Specifically, it reliably penalized contextually degraded responses, ensuring that the evaluation process is sensitive to quality variations. When utilized as structured instructions, text feedback, or training rewards, RubricsTree achieved up to ~66% relative performance gains on the HealthBench benchmark across various model families, including Gemini, GPT, and Qwen. These results underscore the framework’s potential to enhance the evaluation and optimization of personal health agents.

Limitations
The authors acknowledge that while RubricsTree offers a scalable solution, its effectiveness is contingent on the quality of the initial rubric set and the human curation process. Additionally, the reliance on a human-in-the-loop approach may introduce biases based on the expertise of the panel. The paper does not address potential limitations related to the generalizability of the rubrics across diverse healthcare contexts or the computational overhead associated with the adaptive routing mechanism.

Why it matters
RubricsTree represents a significant advancement in the evaluation of personal health agents, providing a scalable and clinically aligned framework that can adapt to evolving healthcare needs. Its ability to yield substantial performance improvements on established benchmarks indicates its potential for real-world applications in healthcare AI. This work lays the groundwork for future research in scalable evaluation methodologies, as published in arXiv cs.AI. The implications extend to enhancing the reliability and effectiveness of AI-driven healthcare solutions, ultimately contributing to better patient outcomes and more equitable healthcare access.