What a voice says before the words do
A speech system can transcribe perfectly and still handle a call badly, because the words were fine and the caller was furious. Notes on why affect belongs in the pipeline, and why a confidence score is not an explanation.
Every voice product eventually meets the same failure. The transcript is perfect. The intent classification is correct. The response is well-formed and entirely wrong, because the caller was three minutes into an escalating problem and the system answered as though they had just said hello.
Words carry what someone is asking for. Prosody carries how much trouble you are in. Humans read the second channel constantly and mostly without noticing — pitch, loudness, rate, voice quality — and it arrives earlier than the sentence does.
Two traditions that rarely talk
There is a cognitive literature describing how people hear emotion: Ekman's categorical account, Russell's valence-arousal plane, Scherer's component process model with its multiple vocal cues. And there is a deep learning literature that classifies emotion from spectrograms with CNNs and LSTMs, and more recently with transformer speech encoders — wav2vec, HuBERT, Whisper — that hold up far better in noise.
The two bodies of work rarely constrain each other, and that is the gap my paper is about. The interesting question is not whether a transformer can beat a benchmark. It is whether the model can be made accountable to the cues a person would have cited.
Why accountability is a shipping requirement
A classifier that outputs "angry, 0.91" and nothing else cannot be deployed into a clinic or a contact centre. Not because the number is wrong, but because there is no way to audit it, no way to argue with it, and no way to explain to the person it was wrong about what happened.
Attention maps and post-hoc explanation methods that point back at the actual signal — this pitch rise, that loudness spike — turn a score into something a human operator can accept or override. That is the difference between a research result and a feature.
The context problem the benchmarks hide
Most speech emotion datasets are acted and studio-clean. IEMOCAP, RAVDESS, EmoDB, CREMA-D are all valuable and none of them sound like a phone call from a moving car.
This matters more than a few points of accuracy. A raised voice on a noisy street is not the same signal as a raised voice in a quiet office — loudness and pitch cues overlap with ambient noise, and a model trained on clean audio will report the overlap as emotion with complete confidence. Context-aware processing is not a refinement here; without it the system is confidently wrong in exactly the environments real calls come from.
Where this lands in a real pipeline
- Preprocess with the acoustic environment as an explicit input, not an assumed constant.
- Combine the cues humans use — pitch, loudness, rate — with learned representations, rather than replacing one with the other.
- Fine-tune a transformer speech encoder on emotional data rather than training from scratch.
- Emit a label with the cue that produced it, so an operator can disagree with the reasoning and not just the score.
- Adapt to individual speakers over time; the baseline for one voice is not the baseline for another.
None of this is exotic. It is mostly a matter of refusing to treat affect as a bolt-on classifier and instead giving it the same standing in the pipeline as transcription — with the same expectation that it can be inspected when it gets something wrong.