Feature-Augmented Transformers for Robust AI-Text Detection Across Domains and Generators

Problem
This preprint addresses the challenge of robust AI-generated text detection across diverse domains and generation pipelines. Existing detectors often struggle with distribution shifts, leading to significant performance degradation when applied to unseen data. The authors highlight the need for a more resilient detection framework that can maintain performance despite variations in text generation sources and styles.

Method
The authors propose a feature-augmented transformer architecture, specifically leveraging DeBERTa-v3-base, enhanced with attention-based linguistic feature fusion. The training process involves using the HC3 PLUS dataset, where a single decision threshold is calibrated to maximize balanced accuracy on a held-out validation set. This threshold is then applied uniformly across various test distributions to evaluate robustness. The model’s performance is assessed on three benchmarks: in-domain on HC3 PLUS, cross-dataset transfer to the M4 benchmark, and external validation on the AI-Text-Detection-Pile. The authors employ multi-seed experiments to ensure stability in their results.

Results
The proposed model achieves a balanced accuracy of 99.5% on the in-domain HC3 PLUS dataset, indicating near-ceiling performance. However, under distribution shifts, the model’s performance drops, revealing its brittleness. The feature-augmented DeBERTa-v3-base+FeatAttn model achieves 85.9% balanced accuracy on the M4 benchmark, outperforming strong zero-shot baselines by up to 7.22 percentage points. Category-level ablation studies indicate that features related to readability and vocabulary significantly enhance robustness against shifts. The results demonstrate that the feature augmentation strategy and the modern DeBERTa architecture provide substantial improvements over traditional BERT/RoBERTa models.

Limitations
The authors acknowledge that while their model shows improved robustness, it remains sensitive to specific types of distribution shifts, which may not be fully captured in the evaluated datasets. Additionally, the reliance on a fixed-threshold protocol may not generalize well to all practical applications, as real-world scenarios often involve dynamic thresholds. The study does not explore the computational cost associated with the feature augmentation process, which could impact scalability.

Why it matters
This work has significant implications for the development of AI text detection systems, particularly in applications where robustness to distribution shifts is critical, such as content moderation and misinformation detection. By demonstrating the effectiveness of feature augmentation in enhancing model stability, the findings encourage further exploration of hybrid approaches that combine transformer architectures with linguistic insights. This research sets a precedent for future studies aiming to improve the resilience of AI systems in the face of evolving text generation techniques.

Authors: Mohamed Mady, Johannes Reschke, Björn Schuller
Source: arXiv:2605.03969
URL: https://arxiv.org/abs/2605.03969v1