Chartographer: Counterfactual Chart Generation for Evaluating Vision-Language Models

Problem
This paper addresses the limitations of current chart question-answering (QA) benchmarks, which often allow models to exploit shortcuts or leverage prior knowledge rather than demonstrating genuine visual reasoning capabilities. The authors highlight that existing evaluations do not adequately assess a model’s ability to generalize across different chart representations. This work is presented as a preprint, indicating it has not yet undergone peer review.

Method
The authors introduce Chartographer, a novel framework designed to generate counterfactual charts for evaluating vision-language models (VLMs). The core technical contributions include:

1. Chart Reconstruction: The framework reverse engineers charts into executable code, ensuring high fidelity in reconstruction.
2. Counterfactual Generation: It generates seed-controlled counterfactual variants of charts while maintaining the original chart-question task. This allows for systematic variation of the underlying chart and corresponding answers.
3. Executable QA Logic: The framework derives new answers based on the executable logic of the QA task, enabling a robust evaluation of model performance across varied chart representations.

The authors apply Chartographer to existing chart QA datasets, focusing on proprietary and open-source VLMs. The training compute details are not disclosed, but the methodology emphasizes the generation of diverse chart representations to test model robustness.

Results
The evaluation reveals that VLMs exhibit significant variation sensitivity and often fail to generalize when faced with counterfactual charts. Specifically, the authors report that models that correctly answer the original chart questions struggle with updated charts that necessitate novel visual reasoning pathways. The results indicate that the failure rates are notably high, although specific numerical performance metrics against baseline models are not provided in the abstract. The findings suggest that traditional performance metrics may mask underlying deficiencies in model reasoning capabilities.

Limitations
The authors acknowledge several limitations, including the potential for bias in the generated counterfactual charts and the reliance on existing datasets, which may not cover the full spectrum of chart types. They also note that the framework’s effectiveness is contingent on the quality of the initial chart reconstruction. An additional limitation not explicitly mentioned is the scalability of the approach; generating a diverse set of counterfactuals may require substantial computational resources, which could limit practical applications.

Why it matters
This work has significant implications for the evaluation of VLMs, particularly in the context of visual reasoning tasks. By introducing counterfactual charts, the authors provide a more rigorous framework for assessing model generalization and reasoning capabilities, which is crucial for advancing the field of visual question answering. The findings underscore the need for more robust evaluation methodologies that can reveal hidden failures in model performance, ultimately guiding future research towards developing models that genuinely understand and reason about visual data.

Authors: Yifan Jiang, Dae Yon Hwang, Jesse C. Cresswell, Freda Shi
Source: arXiv:2605.27311
URL: https://arxiv.org/abs/2605.27311v1