Sunbird AI’s recent success in the Kinyarwanda Kaggle competition highlights advancements in Automatic Speech Recognition (ASR) for low-resource languages. We detail our systematic investigation into optimizing the Whisper-large-v3 model, focusing on the critical role of data volume in achieving robust ASR performance for Kinyarwanda.
Data Scaling Methodology
Our objective was to quantify the relationship between training data volume and ASR model performance. We trained the Whisper-large-v3 model on various subsets of our meticulously curated Kinyarwanda dataset. Training audio splits were established, ranging from a minimal 1 hour to approximately 1400 hours. Each model was independently trained and rigorously evaluated against a consistent “devtest” split to ensure unbiased performance comparison.
Performance Analysis
The experimental results consistently demonstrated a direct correlation between increased training data and improved ASR performance. As shown in accompanying figures, both the Word Error Rate (WER) and Character Error Rate (CER) exhibited a clear downward trend with larger training datasets. Notably, we achieved a sub-10% WER (9.82%) with the 200-hour dataset. This performance was closely aligned with results from the 150-hour (10.21%) and 100-hour (10.90%) experiments, suggesting diminishing returns on performance gains beyond certain data volumes. This highlights an optimal point for balancing data acquisition and computational efficiency.
Computational Strategy and Model Convergence
Model training was primarily executed on 3xH100 GPUs, with select experiments also utilizing A100 GPUs. A key aspect of our methodology involved the precise optimization of training scripts for each experiment. Our goal was to achieve optimal model performance at the point of convergence, preventing overfitting,where the model memorizes training data rather than generalizing, and maximizing its ability to perform on unseen data.
Data illustrating the Word Error Rate evolution during training confirmed that experiments with lower training durations converged at a faster rate. This highlights a crucial operational trade-off: while more data typically yields superior performance, it concurrently necessitates increased training time and computational resources. Despite variations in training data volume, the inference time for a model trained on 1000 hours is similar to that of a model trained on 1 hour, and generally increases with the length of the audio as expected.
Future Work and Real-World Application
While these results are promising, it is important to contextualize the evaluation. The test data utilized was devoid of environmental noise, meaning the reported metrics reflect performance in a relatively controlled environment. Additionally, our current evaluation did not extensively cover the full spectrum of spoken communication types, such as informal conversational speech, public announcements, or call center dialogues. For instance, ASR models for other African languages, including Ugandan languages, that achieve WER < 0.1 on clean test data can exhibit WER > 0.3 in real-world applications due to variations in intonation, vocabulary, and ambient conditions.
These observations underscore the ongoing challenges in deploying ASR models for complex, real-world scenarios. Our success in the Kaggle competition is a strong testament to Sunbird AI’s technical capabilities and strategic approach to ASR development for Kinyarwanda. We are committed to leveraging these insights to further enhance model robustness, expand data diversity, and ultimately bridge the gap between benchmark performance and impactful real-world applications for African languages, thereby contributing to broader digital inclusion efforts.
Written by Nimpamya Janat Namara, Communications and Engagement Lead at Sunbird AI.