Speech Emotion Recognition: When Signal Processing Still Beats Deep Learning

Speech carries more than words. The same sentence can be a question, an accusation, or a joke depending entirely on how it is said, and none of that survives transcription. Systems that only read the text of speech discard most of the signal a human listener uses.

Recovering that discarded signal is the problem of speech emotion recognition, and it is a useful lens on a broader question: when does the classical signal processing approach still beat throwing a deep network at the raw waveform?

This piece relates to earlier work of mine on emotional expression detection in spoken language and on Bangla isolated speech recognition.

Why this is harder than it sounds

Emotion in speech is carried by prosody — pitch contour, energy, rhythm, voice quality — and these interact with everything else in ways that resist clean separation.

Speaker variation dominates. Two people expressing the same emotion produce acoustically different signals, and the difference between speakers is frequently larger than the difference between emotions. A model that learns speakers rather than emotions will score well on a test set that shares speakers with training and collapse on anyone new. Speaker-independent evaluation is the only honest evaluation, and a surprising amount of published work does not do it.

Culture and language shape expression. Prosodic patterns signalling politeness, deference, or anger differ substantially across languages. A model trained on one language does not transfer cleanly, and the assumption that emotion is universal in its acoustic realisation does not hold.

Then there is the data problem, which is worse than the modelling problem. Most datasets consist of actors performing emotions on request. Acted emotion is exaggerated and prototypical; real emotion is subtle, mixed, and often ambiguous even to human annotators. Models trained on acted data learn to recognise performance. Systems that work well in the lab and poorly in deployment usually failed here rather than in the architecture.

Where classical signal processing still earns its place

The reflex is to feed a spectrogram to a convolutional network. Sometimes that is right. Often it is not, and the reasons are worth knowing.

Adaptive decomposition suits non-stationary signals

Speech is non-stationary — its statistical properties change continuously. The short-time Fourier transform handles this by assuming stationarity within a fixed window, which forces a fixed trade-off between time and frequency resolution across the whole signal.

Empirical wavelet transform approaches take a different route: analyse the signal’s own spectrum and build filters adapted to where its energy actually sits, rather than imposing a fixed basis. For signals whose informative content is unevenly distributed in frequency — which speech is — this can isolate the relevant components more cleanly than a fixed decomposition.

The practical benefit is not usually a higher ceiling on accuracy. It is that the resulting features are lower-dimensional and more interpretable, which matters when you have limited data.

Small data favours engineered features

Deep learning’s advantage grows with data volume. Emotional speech datasets are small — often a few thousand utterances from a few dozen speakers. That is well below where end-to-end learning reliably outperforms good features plus a well-regularised classifier.

With limited data, a support vector machine on carefully chosen prosodic and spectral features frequently matches or beats a network trained on the same set, and it does so with far less variance across runs. It also fails more predictably, which matters more in deployment than a marginal accuracy difference.

Edge deployment has hard constraints

If inference has to run on a device, the calculation changes entirely. Memory, power, and latency budgets are fixed and small. A feature extraction pipeline plus a compact classifier can run in a fraction of the resources a competitive network needs.

The privacy dimension reinforces this. Speech is biometric data — a forensic and integrity concern as much as a privacy one. Processing on-device rather than shipping audio to a server is meaningfully better for the person being recorded, and it is often the difference between a system that is deployable in a regulated environment and one that is not. Any architecture that only works in the cloud has made a consequential decision about user data, whether or not anyone framed it that way.

Where deep learning clearly wins

To be even-handed: with enough data, learned representations beat engineered ones, and self-supervised speech models pre-trained on very large unlabelled corpora have changed what is achievable. Fine-tuning such a model on a small emotional dataset routinely outperforms classical pipelines, because the representation was learned from data volumes no individual project can assemble.

The honest position is therefore conditional. If you can access a strong pre-trained speech model and you have labelled data to fine-tune on, use it. If you are working in a low-resource language with no such model, with a small dataset, or under tight edge constraints, classical approaches remain competitive and are considerably easier to reason about.

The low-resource language problem

Most of this technology was built for English, and the gap for other languages is not merely one of accuracy.

Working on Bangla speech recognition made the scale of it concrete. Bangla has hundreds of millions of speakers and a fraction of the annotated corpora, pre-trained models, and pronunciation resources that English enjoys. That absence compounds: no data means no pre-trained models, which means every project starts from nothing, which means fewer projects, which means no data.

Building and releasing datasets in under-resourced languages is unglamorous work with modest citation returns, and it is probably higher-leverage than another incremental architecture on a well-served language. The bottleneck in most of the world is not model capacity.

What I would tell someone starting here

Evaluate speaker-independently or do not report the number. Be sceptical of results on acted datasets. Establish a classical baseline before reaching for a network, because you need to know what the extra complexity bought. And decide early whether inference runs on a device or in a cloud, because that constraint determines the architecture more than accuracy targets do.