KuralHubGuide
Speech emotion recognition: a practical guide
Speech emotion recognition (SER), also called emotion recognition from speech or voice emotion detection, predicts how someone feels from the sound of their voice. This guide explains how emotion detection models work, which datasets and pretrained models to start with, and what the KuralHub benchmark found across 29 languages.
By Luxshan Thavarasa and the KuralHub authors, based on the Interspeech 2026 paper.
Contents
What is speech emotion recognition?
Speech emotion recognition is the task of labelling a stretch of speech with an emotion, usually one of a small set such as anger, happiness, sadness, fear or neutral. It is part of affective computing and matters for any system that talks with people: voice assistants, call-centre analytics and health monitoring.
SER is one branch of emotion analysis. Text-based sentiment analysis reads the words, and facial emotion recognition looks at the face. SER listens to how something is said: pitch, loudness, timing and voice quality. The same sentence can sound calm or furious, and an emotion recognition model for speech is trained to hear the difference.
How an emotion detection model works
Every SER system turns audio into features and then classifies those features. Approaches have moved through three stages.
- Hand-crafted acoustic features. Early systems measured pitch, energy, duration and spectral properties, then used a standard classifier.
- Deep learning on spectral features. Later systems learned from MFCCs or log-Mel spectrograms with CNNs, RNNs or hybrids of the two.
- Self-supervised speech encoders. Current systems start from a model pretrained on large amounts of unlabelled speech, such as HuBERT, wav2vec 2.0, WavLM or Whisper, and add a small emotion classifier on top. The encoder is either kept frozen, fully fine-tuned, or fine-tuned with a parameter-efficient method.
KuralHub uses the third approach with the encoder frozen: audio at 16 kHz goes through the pretrained model, the last-layer features are averaged over time, and a two-layer head predicts the emotion. Freezing keeps training cheap and makes models easy to compare. The benchmark setup has the details and the code is open.
Emotion labels
Most datasets use categories. The core set shared by many corpora is anger, happiness, sadness, fear and neutral, often extended with disgust, surprise, calm or boredom. Some datasets instead use dimensions, rating each clip for arousal (how activated the speaker sounds) and valence (how positive). Because label sets differ between datasets, cross-dataset work usually maps everything to a shared subset. KuralHub uses the five core emotions.
Speech emotion datasets
Well-known datasets include IEMOCAP, RAVDESS and MSP-Podcast for English, EmoDB for German and CASIA for Mandarin. They differ in number of speakers, recording conditions, how emotion was elicited (acted or spontaneous) and labelling scheme, so pick one that matches your use case.
Outside a few major languages, data is hard to find. The KuralHub catalog lists 90 speech emotion datasets for the 70 most widely spoken languages, with emotion labels, size, licence, access link and BibTeX for each. 45 of those languages have at least one dataset and 25 have none we could find.
Popular language pages: English · Chinese · Hindi · Arabic · Spanish · French · German · Bengali · Tamil · Urdu · Japanese · Korean · all 70 languages.
Which pretrained model to use
The KuralHub benchmark trained the same classifier on 11 frozen encoders across 33 datasets in 29 languages. In short:
- Whisper had the highest mean F1 (0.70 for both whisper-small and whisper-large).
- HuBERT-base came close (0.68) at a fraction of the size, followed by WavLM (0.63 to 0.64).
- wav2vec 2.0 models generalised worst, and the larger wav2vec 2.0 checkpoints did worse than the base one.
- No model won everywhere. HuBERT was best for nine languages and WavLM for English and Swahili.
- Bigger was not better. The log of parameter count was negatively correlated with F1 (r = −0.18, p = 0.0007).
So a good default emotion recognition model is Whisper or HuBERT-base, but check the results for your language before committing.
How to evaluate
Emotion datasets are often imbalanced, with far more neutral clips than fearful ones, so plain accuracy can flatter a model. Common metrics are:
- Weighted accuracy (WA): the share of clips classified correctly.
- Unweighted average recall (UAR): recall averaged over emotions, each counted equally. This is the usual choice for imbalanced data.
- Macro-F1: F1 averaged over emotions, each counted equally.
Report scores per emotion too. In KuralHub, fear was the hardest emotion (F1 0.38) and anger (0.54) and neutral (0.52) were the easiest. The results file has accuracy, macro recall (UAR), macro-F1 and per-emotion F1 for every run.
What makes SER hard
- Language and culture. People express emotion differently across languages and cultures. Mean accuracy across all models and languages in KuralHub was 0.52, ranging from 0.755 for Mandarin to 0.275 for Kannada.
- Language structure. Dravidian languages (Tamil, Telugu, Kannada) were hard for every model. The paper links this to their agglutinative morphology and retroflex consonants, which need fine-grained temporal modelling.
- More data is not enough. High-resource languages such as French, Spanish and Russian scored lower than several low-resource ones.
- Acted versus natural speech. Recording conditions explained about 40% of the variance in performance. The acted Amharic ASED dataset scored 24% higher than spontaneous datasets with similar resources.
- Scarce data. Most languages have one small dataset or none at all.
Getting started
- Find a dataset for your language in the catalog and check its licence and access route.
- Build a baseline: a frozen Whisper or HuBERT-base encoder with a small classification head. The KuralHub code does this out of the box.
- Report UAR or macro-F1 per emotion, not just accuracy.
- Compare with the KuralHub numbers for the same dataset.
- If your dataset is missing from the catalog, add it so others can find it.
Frequently asked questions
What is speech emotion recognition?
Speech emotion recognition (SER) is the task of predicting a speaker's emotion, such as anger, happiness, sadness, fear or a neutral state, from the sound of their voice. It is also called emotion recognition from speech or voice emotion detection.
How is speech emotion recognition different from sentiment analysis?
Sentiment analysis usually reads text and decides whether it is positive or negative. Speech emotion recognition listens to how something is said, using cues such as pitch, energy, timing and voice quality, and usually predicts specific emotions rather than a polarity.
What is the best model for speech emotion recognition?
In the KuralHub benchmark of 11 pretrained speech models across 29 languages, Whisper had the highest mean F1 (0.70), followed by HuBERT-base (0.68) and WavLM (0.63 to 0.64). No model was best for every language, and larger models were not better, so check results for your language first.
Which dataset should I use for speech emotion recognition?
It depends on the language, licence and whether you need acted or natural speech. Well-known English datasets include IEMOCAP, RAVDESS and MSP-Podcast. The KuralHub catalog lists 90 datasets for 70 languages with emotion labels, size, licence and access links.
How accurate is speech emotion recognition?
It varies widely by language and dataset. In the KuralHub benchmark the mean test accuracy over 11 models and 29 languages was 0.52. The best single run reached 0.98 on the Mandarin ESD dataset, but the best Tamil run reached only 0.44.
Is there a speech emotion recognition benchmark across many languages?
Yes. KuralHub (Interspeech 2026) benchmarks 11 self-supervised models on 33 datasets in 29 languages. EmoBox and SER Evals are other multilingual SER benchmarks.
Does speech emotion recognition work for low-resource languages?
Sometimes. In KuralHub, lower-resource languages such as Persian, Arabic, Odia, Amharic and Quechua outperformed several European languages, while Dravidian languages such as Tamil, Telugu and Kannada were hard for every model.
Citation
If this guide or the KuralHub data helped your work, please cite:
Thavarasa, L., Thevakumar, J., Sivatheepan, T., Thayasivam, U. (2026) KuralHub: Exposing Typological Capability Frontiers in Multilingual Speech Emotion Recognition. Proc. Interspeech 2026, 4673-4678, doi: 10.21437/Interspeech.2026-3502