CC-OCR V2: Benchmarking Large Multimodal Models for Literacy in Real-world Document Processing

Problem
This paper addresses the inadequacy of existing benchmarks for evaluating Large Multimodal Models (LMMs) in real-world Optical Character Recognition (OCR) tasks. Current benchmarks often misalign with practical applications, focusing on homogeneous conditions and neglecting the complexities encountered in enterprise document processing. The authors highlight that while LMMs have demonstrated strong performance in controlled settings, their effectiveness in diverse, real-world scenarios remains underexplored. This work is presented as a preprint, indicating it has not yet undergone peer review.

Method
The authors introduce CC-OCR V2, a novel benchmark designed specifically for real-world document processing. This benchmark encompasses five major OCR-centric tracks: text recognition, document parsing, document grounding, key information extraction, and document question answering. It consists of 7,093 high-difficulty samples that include challenging and corner cases often overlooked in prior benchmarks. The evaluation methodology involves extensive experiments on 14 advanced LMMs, assessing their performance across the defined tracks. The authors provide a comprehensive evaluation toolkit alongside the dataset, facilitating reproducibility and further research.

Results
The experiments reveal that even state-of-the-art LMMs exhibit significant performance degradation when evaluated on the CC-OCR V2 benchmark compared to their performance on existing benchmarks. Specific performance metrics are not disclosed in the abstract, but the authors emphasize that the gap between benchmark performance and real-world applicability is substantial. This indicates that current models are not adequately equipped to handle the complexities of practical document processing tasks, underscoring the need for improved model architectures and training methodologies.

Limitations
The authors acknowledge that their benchmark, while comprehensive, may still not capture all possible real-world scenarios, particularly in highly specialized domains. They also note that the performance metrics provided may vary based on the specific configurations and training regimes of the LMMs tested. An additional limitation not explicitly mentioned is the potential for dataset bias, as the samples may not represent the full diversity of document types encountered in practice.

Why it matters
The introduction of CC-OCR V2 has significant implications for the development and evaluation of LMMs in document processing. By highlighting the performance gaps in current models, this work encourages researchers to focus on enhancing model robustness and adaptability to real-world conditions. The benchmark serves as a critical resource for future research, pushing the boundaries of OCR capabilities and fostering advancements in multimodal learning. The release of the dataset and evaluation toolkit also promotes collaboration and innovation within the community, potentially leading to breakthroughs in document literacy applications.

Authors: Zhipeng Xu, Junhao Ji, Zulong Chen, Zhenghao Liu, Qing Liu, Chunyi Peng, Zubao Qin, Ze Xu et al.
Source: arXiv:2605.03903
URL: https://arxiv.org/abs/2605.03903v1