Text Analytics Evaluation Framework: A Case Study on LLMs and Social Media

Problem
This preprint addresses a significant gap in the evaluation of large language models (LLMs) when applied to practical data analysis scenarios, specifically in processing long sequences of unstructured documents like social media posts. While LLMs have shown strong performance across various NLP tasks, their effectiveness in handling complex, aggregated text data remains underexplored. The authors aim to empirically assess LLMs’ capabilities in semantic understanding and reasoning through a structured evaluation framework.

Method
The authors propose a question-based evaluation framework consisting of 470 manually curated questions that target LLMs’ semantic comprehension and reasoning skills. This framework is applied to diverse Twitter datasets, focusing on tasks such as sentiment analysis, hate speech detection, and emotion recognition. The evaluation emphasizes the impact of input scale and data complexity on model performance. The authors analyze the performance of various LLMs, particularly noting the degradation in effectiveness as task complexity increases—from basic semantic identification to more intricate operations like comparison and numerical calculations. The study also highlights a critical performance drop when input sizes exceed 500 instances, particularly for open-weight models.

Results
The findings indicate that LLM performance is significantly influenced by both the scale of input data and the complexity of the tasks. For instance, in multi-label scenarios, the models exhibited a marked decline in accuracy compared to simpler tasks. The authors report that as task complexity escalates, performance diminishes progressively, with substantial drops observed in numerical tasks. Specific performance metrics are not disclosed in the abstract, but the qualitative assessment suggests that LLMs struggle with rigorous quantitative analysis over large text collections, particularly when faced with intricate reasoning requirements.

Limitations
The authors acknowledge several limitations, including the reliance on manually curated questions, which may introduce bias or limit the generalizability of the findings. They also note that the performance degradation observed in open-weight models may not be representative of all LLM architectures. Additionally, the study does not explore the potential for fine-tuning or other optimization strategies that could mitigate the identified performance issues. An obvious limitation not discussed is the lack of comparison with other state-of-the-art models or frameworks that might provide a more comprehensive understanding of LLM capabilities in this context.

Why it matters
This work has significant implications for the deployment of LLMs in real-world applications involving large-scale text data analysis. By identifying critical architectural bottlenecks and performance limitations, the study provides a foundation for future research aimed at enhancing LLM capabilities in complex reasoning tasks. The insights gained from this evaluation framework can inform the design of more robust models and evaluation methodologies, ultimately improving the reliability of LLMs in practical data analytics scenarios.

Authors: Yuefeng Shi, Nedjma Ousidhoum, Jose Camacho-Collados
Source: arXiv:2605.21338
URL: https://arxiv.org/abs/2605.21338v1