Conceptors for Semantic Steering

Problem
This paper addresses the limitations of existing activation-based steering methods for large language models (LLMs), which typically reduce concepts to single directional vectors. This reduction neglects the geometric complexity of concepts, leading to suboptimal steering performance. The authors propose a novel approach using conceptors—soft projection matrices that capture the full multidimensional subspace of bipolar concepts—allowing for more nuanced control over LLM behavior at inference time. This work is presented as a preprint and has not yet undergone peer review.

Method
The core technical contribution is the introduction of conceptors, which are derived from pooled activations across both poles of a bipolar concept. This approach enables the preservation of the multidimensional subspace associated with each concept. The authors provide a geometric analysis demonstrating that the bipolar subspace strictly subsumes the traditional single-vector baseline. They also introduce a conceptor quota, a parameter-free diagnostic tool that predicts concept separability with high Pearson correlation coefficients (up to r=0.96) across three instruction-tuned models and three semantic dimensions. The paper further explores the compositionality of conceptors through closed-form Boolean algebra operations (AND, OR, NOT) on thematically related sub-concepts. The evaluation is conducted across a systematic five-axis design space, comparing conceptor steering against additive baselines.

Results
The results indicate that conceptor steering matches or outperforms traditional additive baselines in scenarios where concept subspaces are multidimensional. Specifically, the authors report that conceptors produce significantly fewer degenerate outputs compared to single-direction steering methods. While exact numerical performance metrics are not disclosed in the abstract, the qualitative improvements suggest a substantial enhancement in steering efficacy, particularly in complex semantic tasks.

Limitations
The authors acknowledge that their approach may not generalize to all types of concepts, particularly those that are inherently unipolar or less complex. Additionally, the reliance on pooled activations may introduce noise, potentially affecting the quality of the conceptors. The paper does not address the computational overhead associated with estimating the conceptors, which could be a concern in resource-constrained environments. Furthermore, the evaluation is limited to three instruction-tuned models, which may not represent the broader landscape of LLM architectures.

Why it matters
This work has significant implications for the field of LLM steering and control. By providing a geometrically principled method for concept representation, conceptors enable more effective and safer steering of LLMs, reducing the risk of generating undesirable outputs. The ability to perform Boolean operations on concepts opens new avenues for compositional reasoning in LLMs, potentially enhancing their interpretability and usability in complex applications. This research lays the groundwork for future explorations into multidimensional concept representations and their applications in various NLP tasks.

Authors: Ilias Triantafyllopoulos, Young-Min Cho, Ren Tao, Miranda Muqing Miao, Sunny Rai, Lyle Ungar, Sharath Chandra Guntuku, Neville Ryant et al.
Source: arXiv:2605.04980
URL: https://arxiv.org/abs/2605.04980v1