PilotTTS: A Disciplined Modular Recipe for Competitive Speech Synthesis

Problem
This paper addresses the challenge of developing competitive text-to-speech (TTS) systems without the need for extensive proprietary datasets and complex architectures, which often pose significant barriers for resource-constrained research teams. The authors present PilotTTS, a lightweight autoregressive TTS system that is designed to operate effectively with only 200K hours of data, processed using open-source tools. This work is presented as a preprint and has not yet undergone peer review.

Method
PilotTTS introduces a disciplined modular approach to TTS architecture and data processing. The core technical contributions include:

1. A reproducible multi-stage data processing pipeline that encompasses quality assessment, label annotation, and filtering, ensuring high-quality input data.
2. A compact model architecture that utilizes Q-Former-based conditioning to effectively decouple speaker identity from speaking style through cross-sample paired training. This allows the model to generalize across different speaking styles and speaker identities. The architecture is autoregressive, facilitating the generation of speech in a sequential manner. The training compute details are not disclosed, but the authors emphasize the efficiency of their approach compared to traditional methods requiring larger datasets.

Results
PilotTTS demonstrates competitive performance on the Seed-TTS Eval benchmark, achieving a word error rate (WER) of 1.50% on the English test set and a character error rate (CER) of 0.87% on the Chinese test set. Additionally, it achieves the highest speaker similarity scores of 0.862 for English and 0.815 for Chinese, outperforming existing systems that were trained on significantly larger datasets. These results indicate that PilotTTS not only matches but exceeds the performance of more complex models, showcasing the effectiveness of its minimalist design and rigorous data engineering.

Limitations
The authors acknowledge that while PilotTTS achieves competitive results, its performance may still be limited by the relatively small dataset size compared to state-of-the-art systems that utilize millions of hours of data. Additionally, the paper does not address potential biases in the training data or the generalizability of the model across diverse languages and accents beyond those explicitly tested. The reliance on open-source tools for data processing may also introduce variability in data quality that could affect performance.

Why it matters
The implications of PilotTTS are significant for the field of speech synthesis, particularly for researchers and developers in resource-constrained environments. By providing a modular and reproducible framework, along with a complete data processing pipeline and pretrained weights, the authors lower the barrier to entry for developing competitive TTS systems. This work encourages further exploration of lightweight architectures and efficient data utilization, potentially leading to advancements in voice cloning, emotion synthesis, and dialect generation. The release of the code and data pipeline fosters collaboration and innovation in the TTS community.

Authors: Bowen Li, Shaotong Guo, Zhen Wang, Yang Xiang, Mingli Jin, Yihang Lin, Jiahui Zhao, Weibo Xiong et al.
Source: arXiv:2605.27258
URL: https://arxiv.org/abs/2605.27258v1