LLM Classification Is Feature Engineering

Recent research by Han et al., Balek et al., and Malberg, Mosca & Groh explores the application of large language models (LLMs) as classifiers, particularly in the context of irony detection using the SemEval 2018 Task 3 dataset. This dataset comprises a total of 4,618 tweets, with 3,834 allocated for training and 784 for testing.

The study reports a true positive rate (TPR) of 0.965 for the LLM classifier, indicating a high level of accuracy in identifying ironic tweets. The false negative rate (FNR) is notably low at 0.035, suggesting that the model effectively minimizes misclassifications. The Brier score for the LLM classifier stands at 0.259, reflecting its probabilistic predictions.

When compared to traditional methods, the LLM classifier's performance is competitive. For instance, logistic regression achieves a Brier score of 0.175 and an F1 score of 0.747, while the LLM classifier matches this F1 score. However, when LLM features are integrated into logistic regression, the Brier score improves to 0.131, and the F1 score increases to 0.768. Similarly, rule-based features yield a Brier score of 0.127 and an F1 score of 0.779, outperforming the LLM classifier alone.

The research also benchmarks other models, with the SVM combined with tf-idf yielding an F1 score of 0.589, and the THU_NGN and NTUA-SLP models achieving F1 scores of 0.705 and 0.786, respectively. The LLM's hard label F1 score is reported at 0.747, with a confidence interval of 0.712 to 0.778. Additionally, when combined with logistic regression, the LLM's F1 score reaches 0.779, with a confidence interval of 0.746 to 0.81.

The authors emphasize the challenges in optimizing LLMs for classification tasks, stating that while it requires significant effort, the potential impact of successfully deploying these models as classifiers is substantial. Upcoming research papers are anticipated to further explore these findings at conferences such as ICML 2024 and ECML PKDD 2024.