MATCHA: Matching Text via Contrastive Semantic Alignment

Problem
This paper addresses the inadequacies of existing evaluation metrics for large language models (LLMs), specifically token-overlap scores (e.g., ROUGE) and embedding-based measures (e.g., BERTScore). The authors highlight that these metrics often yield misleading results, assigning similar scores to semantically contradictory texts, which can obscure critical errors in model outputs. This work is presented as a preprint and has not undergone peer review.

Method
The authors propose MATCHA (Matching Text via Contrastive Semantic Alignment), a novel evaluation metric that integrates two key components: (1) a measure of semantic proximity to a reference text and (2) a measure of distance from an adversarially generated counterfactual that contradicts the reference. This dual-view approach allows MATCHA to reward semantic agreement while penalizing contradictions, thereby providing a more nuanced evaluation of text quality. The metric is evaluated across eight public benchmarks, including tasks such as question-answering, image caption generation, natural language inference, summarization, and semantic textual similarity. The authors do not disclose specific details regarding the architecture or training compute used for the adversarial generation of counterfactuals.

Results
MATCHA demonstrates significant improvements over established metrics. On the TruthfulQA dataset, which lacks a training set for embedding-based metrics, MATCHA achieves an 18.38% improvement over ROUGE-L and a 20.82% improvement over BERTScore in terms of matching texts with a reference. Across the eight benchmarks, MATCHA consistently outperforms popular metrics, with quantitative comparisons and qualitative human assessments validating its effectiveness. Notably, MATCHA outperforms 23 embedding models, including leading state-of-the-art metrics, in accurately distinguishing correct from incorrect statements based solely on a reference.

Limitations
The authors acknowledge that while MATCHA improves upon existing metrics, it may still be sensitive to the quality of the reference texts used for evaluation. Additionally, the reliance on adversarially generated counterfactuals may introduce variability depending on the generation process. The paper does not address potential computational overhead associated with the dual-view evaluation, which could limit its scalability in real-time applications.

Why it matters
The introduction of MATCHA has significant implications for the evaluation of LLMs, particularly in applications where semantic accuracy is critical. By addressing the shortcomings of traditional metrics, MATCHA provides a more reliable framework for assessing model outputs, which can lead to improved model training and deployment strategies. This work encourages further exploration of contrastive evaluation methods and may inspire the development of additional metrics that prioritize semantic integrity in language generation tasks.

Authors: Siran Li, Ece Sena Etoglu, Carsten Eickhoff, Seyed Ali Bahrainian
Source: arXiv:2605.27345
URL: https://arxiv.org/abs/2605.27345v1