DRScaffold: Boosting Dense-Scene Reasoning in Lightweight Vision Language Models

Problem
This paper addresses the limitations of lightweight vision-language models (VLMs) in dense-scene reasoning, a critical capability for interpreting complex visual environments. Despite their competitive performance on standard benchmarks, these models struggle with tasks requiring multi-step inference involving multiple objects, attributes, and relations. The authors highlight a lack of explicit grounding in existing training signals, which leads to fluent but visually unanchored reasoning chains. The work is presented as a preprint, indicating it has not yet undergone peer review.

Method
The authors introduce DRBench, a novel benchmark comprising 14,573 questions across 2,943 images, categorized into five task types that reflect three progressive reasoning layers. To enhance dense-scene reasoning, they propose DRScaffold, a supervised fine-tuning framework that decomposes the supervision target into four causally ordered stages. This approach enforces grounded reasoning without necessitating architectural changes to the existing lightweight VLMs. The framework leverages structured supervision to guide the model through the reasoning process, effectively linking reasoning steps to the underlying visual entities and relations.

Results
Experiments conducted on three lightweight VLMs demonstrate significant performance improvements on DRBench. Notably, the Qwen2.5-VL-3B model, when fine-tuned with DRScaffold, outperforms the frozen Qwen2.5-VL-32B model on the same benchmark. The authors report that DRScaffold not only enhances performance on DRBench but also maintains or improves results on general-purpose benchmarks, indicating that structured supervision can effectively substitute for increased model scale in dense-scene reasoning tasks.

Limitations
The authors acknowledge that while DRScaffold improves performance in dense-scene reasoning, it may not generalize to all types of reasoning tasks outside the scope of DRBench. Additionally, the reliance on structured supervision may limit the model’s adaptability to novel reasoning scenarios not covered in the training data. The paper does not address potential computational overhead introduced by the multi-stage supervision process, which could impact training efficiency.

Why it matters
The implications of this work are significant for the development of lightweight VLMs capable of robust reasoning in real-world applications. By demonstrating that structured supervision can enhance reasoning capabilities without increasing model size, this research paves the way for more efficient models that can operate effectively in cluttered environments. The introduction of DRBench also provides a valuable resource for future research, enabling the evaluation and comparison of reasoning capabilities across various models.

Authors: Xinrui Shi, Kai Liu, Ziqing Zhang, Jianze Li, Anqi Li, Yulun Zhang
Source: arXiv:2605.26038
URL: https://arxiv.org/abs/2605.26038v1