Coupled but Late: Turn-Taking Between Full-Duplex Speech Models in Unscripted Dialogue

Problem

This work addresses a gap in understanding the timing dynamics of full-duplex speech models during unscripted dialogue. The authors highlight that existing models do not adequately capture the nuances of turn-taking timing, which is critical for improving conversational AI systems. The paper is a preprint and has not undergone peer review.

Method

The authors employ two instances of the PersonaPlex-7B architecture to model turn-taking in unscripted conversations. The data consists of audio tokens exchanged between the models during dialogue. A floor-transfer rule is applied to both models and the Switchboard dataset to facilitate the analysis of turn-taking dynamics. The timing of floor changes is measured, with a focus on the median timing of these changes during interactions.

Results

The median floor change timing observed in the models ranges from 400 to 560 ms, which is significantly slower than the 137 ms median timing for human interlocutors. Additionally, the analysis reveals that the last 120 ms of a partner's turn holds only 1% of transfers compared to human projections, indicating a substantial discrepancy in timing strategies between the models and human speakers.

Limitations

The authors note that the coupling of timing is disrupted when speakers are re-paired, which may affect the reliability of the results. Furthermore, the timing of the model's turn-end projections does not align with human expectations, suggesting that the models may not fully replicate human conversational dynamics.

Why it matters

Understanding the timing of turn-taking in dialogue systems is crucial for developing more natural and effective conversational agents. The findings highlight the need for further research into timing mechanisms in AI models, which could lead to improvements in human-computer interaction and more sophisticated dialogue systems.