Less Back-and-Forth: A Comparative Study of Structured Prompting

Problem
This preprint addresses the gap in understanding how structured prompting can enhance the performance of large language models (LLMs) in open-ended tasks. Specifically, it investigates the impact of different prompting strategies on the quality of responses generated by LLMs, as underspecified prompts often lead to suboptimal answers and necessitate additional user interaction. The study aims to provide empirical evidence on whether structured prompts can mitigate these issues.

Method
The authors conducted a comparative study involving three distinct prompting conditions: (1) raw prompts, (2) checklist-improved prompts, and (3) clarifying-question prompts. The evaluation was performed across four task types: summarization, planning, explanation, and coding. Three LLM systems were utilized for the experiments: ChatGPT, Claude, and Grok. Outputs from each model were assessed using a unified rubric that encompassed four criteria: task completion, correctness, compliance, and clarity. The checklist-improved prompts were designed to guide the model more effectively, while the other two conditions served as baselines. The study also measured the average token usage for each prompting strategy to evaluate the quality-effort tradeoff.

Results
The results indicated that checklist-improved prompts significantly outperformed both raw prompts and clarifying-question prompts. The mean rubric score for checklist prompts was 7.50 out of 8, compared to 5.67 for raw prompts and 6.67 for clarifying-question prompts. Additionally, the checklist prompts demonstrated a superior quality-effort tradeoff, utilizing fewer average tokens than both competing prompt types. This suggests that structured prompting not only enhances response quality but also reduces the cognitive load on users.

Limitations
The authors acknowledge several limitations, including the potential for bias in the scoring rubric and the limited scope of task types evaluated. The study does not explore the long-term effects of structured prompting on user satisfaction or the adaptability of LLMs to different domains beyond those tested. Furthermore, the reliance on a single scoring rubric may not capture all nuances of response quality, and the results may vary with different LLM architectures or configurations not included in this study.

Why it matters
This research has significant implications for the design of user interactions with LLMs, particularly in applications requiring high-quality outputs with minimal user effort. By demonstrating that structured prompts can enhance the performance of LLMs, the findings encourage further exploration into prompt engineering as a critical component of LLM deployment. This work lays the groundwork for future studies to investigate additional structured prompting techniques and their applicability across diverse tasks and models, potentially leading to more efficient and effective human-AI collaboration.

Authors: Saurav Ghosh, Gabriella Polach, Abdou Sow
Source: arXiv:2605.20149
URL: https://arxiv.org/abs/2605.20149v1