The Shibboleth Effect: Auditing the Cross-Lingual Distributional Skew of Large Language Models

Problem
This study addresses the gap in understanding how large language models (LLMs) exhibit cross-lingual distributional skew, termed the Shibboleth Effect, particularly under adversarial conditions. The authors highlight that existing literature lacks comprehensive audits of LLM behavior across different languages, especially in geopolitical contexts. This work is a preprint and has not undergone peer review, indicating that findings should be interpreted with caution.

Method
The authors develop a multi-agent geopolitical wargame, the Cerulean Sea Crisis, simulating a maritime territorial dispute reflective of Eastern Mediterranean conflicts. Six frontier LLMs—GPT-4o, Llama-4, Mistral-Large, Gemini-3.1-Pro, Qwen3.6-Plus, and DeepSeek-R1—are evaluated in a between-groups experimental design (N = 10 games per arm, K = 5 rounds per game). The manipulation involves the language of play (English vs. Turkish), resulting in 586 validated statements. A zero-shot classifier measures behavioral dispositions along two dimensions: Concession Rate and Coercive Rhetoric. The study identifies two buffering mechanisms: chain-of-thought institutional anchoring and multilingual reinforcement learning from human feedback (RLHF) alignment.

Results
The results reveal heterogeneous effects across models. Llama-4 exhibits a significant increase in coercive rhetoric when playing in Turkish (delta = +0.800, Holm-corrected p = .002). In contrast, Gemini-3.1-Pro shows a substantial decrease in coercive rhetoric (delta = -0.750, p = .005), while DeepSeek-R1 also demonstrates a negative shift (delta = -0.860, p = .006), providing evidence for a buffering mechanism. GPT-4o, however, shows no significant effect (delta = +0.130, p = .614). These findings suggest that cross-lingual behavioral skew is not a universal trait of Western-origin LLMs but is contingent on specific model architectures and training methodologies.

Limitations
The authors acknowledge that their findings are limited by the synthetic nature of the geopolitical scenario, which may not fully capture the complexities of real-world conflicts. Additionally, the study’s reliance on a zero-shot classifier may introduce biases in behavioral assessment. The sample size, while adequate for preliminary insights, may not be sufficient for generalizing results across all LLMs or languages. The authors also note that the effects observed may vary with different adversarial conditions or additional languages not included in this study.

Why it matters
This research has significant implications for the deployment of LLMs in diplomatic and crisis-management contexts, emphasizing the need for careful consideration of cross-lingual biases. The identification of distinct buffering mechanisms provides a foundation for future work aimed at mitigating adverse effects in multilingual applications. Understanding these dynamics is crucial for ensuring the safe integration of LLMs in sensitive environments, as discussed in the context of the Shibboleth Effect, which is further elaborated in the full paper available on arXiv.