Evaluation of Open-Source Models Reveals Trustworthiness and Vulnerabilities

The study evaluated 725 responses from models including Kimi K2.6, Kimi K2.7 Code, GPT 5.5, Claude Opus 4.8, and SWE-1.7. Conducted by researchers Pan and Xu, the analysis focused on 145 politically sensitive questions to assess model behavior. Cognition, the company behind SWE-1.7, claims that models developed from open-source foundations can be trusted if developed with care.

SWE-1.7 demonstrated a refusal rate of just 3.6% in evaluations, significantly lower than the 85% refusal rate of the DeepSeek-R1 model on CCP-sensitive topics. Notably, DeepSeek models echoed inaccurate narratives from the Chinese Communist Party (CCP) four times more than their U.S. counterparts. Furthermore, the propaganda rate for DeepSeek-R1 decreased from 6.8% in Simplified Chinese to 2.4% in Traditional Chinese.

Older models, such as Qwen3-Coder, exhibited 130% more vulnerabilities when evaluated for U.S. government applications, raising concerns about the security of AI-coded software. This follows a May 2026 report by CrowdStrike and Booz Allen Hamilton that highlighted similar vulnerabilities.

Cognition asserts that SWE-1.7 performs comparably or favorably against models from U.S. frontier labs, while also emphasizing that open-source models generally demonstrate more concerning behaviors. The effective date for regulations in China mandating adherence to core socialist values was August 15, 2023, further complicating the landscape for AI model development.

Overall, the findings indicate that while open-source models can be reliable, there are significant risks associated with those influenced by political narratives, particularly from China.