Emotion Detection Pipeline for Television Content

Overview

Emotion Detection Pipeline is a proof of concept for analysing emotional language in television dialogue. Developed for the Content Intelligence Agency, it converts audiovisual material into translated transcript segments with predicted emotion labels and confidence scores.

The project investigates replacing the client’s emotion-classification stage with a fine-tuned RoBERTa-base model. It classifies transcript text; it does not infer emotions directly from faces, voice tone, or a person’s internal state.

Research Publication

This project also contributed to a co-authored research article, When Text Is Not Enough: Structural Limits of Text-Only Transformer-Based Emotion Classification, published in the 2026 Bled eConference proceedings.

The study compares SVM, Bi-LSTM, and RoBERTa on a text-only emotion-classification pipeline. It discusses how translation, class imbalance, and missing conversational context affect the results.

Read the paper (PDF) · DOI.

Problem

The client wanted a more controllable alternative to an external large-language-model classifier. The challenge was both model selection and pipeline integration: transcription errors, translation choices, and missing context can change the meaning that reaches the classifier.

Implementation

Video or audio -> transcription -> optional correction -> English translation -> RoBERTa classification -> CSV output.

The pipeline accepts a local file or YouTube input. The current orchestration selects AssemblyAI for transcription; the repository also contains a Whisper implementation. Optional autocorrection combines language checks with an Ollama-hosted model before translation and classification.

RoBERTa performs the emotion classification locally. That deployment boundary matters: local classification does not make the entire configured pipeline offline, because transcription and optional correction can depend on external services.

The classifier predicts anger, disgust, fear, joy, neutral, sadness, and surprise. It supports batch prediction and returns a confidence score, with an optional full class-probability distribution.

Structured Output

The saved CSV preserves transcript information and adds fields for downstream analysis:

Segment information can support downstream timelines, but the core delivered output is a structured CSV rather than a complete media-analytics dashboard. A confidence score expresses model certainty, not independently measured correctness.

Model Development and Evaluation

Experiments compare traditional classifiers, recurrent networks, and transformer approaches. The project presentation reports F1 approximately 0.755 for the selected RoBERTa model. The averaging method is not specified in that presentation, so this should be read as its reported experiment result rather than a general performance guarantee.

The model card documents 30,409 training and validation sentences and a 792-sentence test set from translated television material. It also describes both client-generated labels and manual annotation. The error-analysis report explicitly identifies mistakes in AI-generated reference labels, making label provenance and quality a limitation of the evaluation.

Explainability and Error Analysis

The project explores Gradient x Input attribution, layer-wise relevance propagation, and token-removal analysis to investigate sensitivity to individual words.

Error analysis groups failures involving missing context, similarity between emotion classes, emotionally charged words, translation errors, and questionable reference labels. Short utterances can support several readings, so a sentence-level label may disagree with the surrounding scene.

The practical lesson is to inspect the full chain: better classification alone cannot recover information lost during transcription, translation, or sentence splitting.

Software and Limitations

The repository provides a command-line pipeline, configuration and logging helpers, classifier interfaces, and automated tests for pipeline stages and services. Saved transcription artifacts and CSV outputs support inspection between stages.

The model card identifies the work as an eight-week proof of concept, with class imbalance and limited domain coverage. Broader use would require an audited evaluation set, clearer separation of label sources, and tests on new programmes and languages. Scene context and temporal aggregation are useful future improvements rather than demonstrated core capabilities.

Implementation References

• Pipeline orchestration and service dependencies.
• Classifier and output fields.
• Model card and project presentation.
• Attribution experiments and error-analysis report.

Tech stack: Python, PyTorch, Hugging Face Transformers, RoBERTa, AssemblyAI, Whisper, Ollama, Pandas, Pytest.