Towards modeling expressed emotions in oral history interviews: Using verbal and nonverbal signals to track personal narratives

Research output: Contribution to journalArticleAcademicpeer-review

1 Citation (Scopus)

Abstract

The article aims to model the verbal and prosodic features of emotional expression in interviews to investigate the potential for synergy between scholarly fields that have the narrative as object of study. Using a digital collection of oral history interviews that contains narrative aspects addressing war and violence in Croatia, we analyzed emotional expression through the words spoken, and through the pitch, vocal effort, and pause duration in the speech signal. The findings were correlated with the linear structure of interviews as well as question type. Our analysis indicates that the weight of emotion words for the overall expressed emotion is stronger in later interview parts as well as after open questions and meaning questions. Similar patterns were found for pitch and pause duration, but not for vocal effort. Although the verbal expression of emotions was somewhat correlated to pause duration, the hypothesized correlation between the verbal and nonverbal features was not confirmed. The research also shows that the various expressive layers in the interviews as well as the relations between them are a suited basis for computational modeling that may help track emotional personal narratives in interview collections. Additional research is needed to further develop the framework for the automated analysis of verbal and nonverbal cues to automatically generate annotations to be used for exploring spoken word collections.
Original languageUndefined
Pages (from-to)621-636
Number of pages16
JournalLiterary and linguistic computing
Volume29
Issue number4
DOIs
Publication statusPublished - Dec 2014

Keywords

  • METIS-307450
  • IR-93345
  • EWI-25354

Cite this

@article{051f8c928e93434d84eff28d2df889e0,
title = "Towards modeling expressed emotions in oral history interviews: Using verbal and nonverbal signals to track personal narratives",
abstract = "The article aims to model the verbal and prosodic features of emotional expression in interviews to investigate the potential for synergy between scholarly fields that have the narrative as object of study. Using a digital collection of oral history interviews that contains narrative aspects addressing war and violence in Croatia, we analyzed emotional expression through the words spoken, and through the pitch, vocal effort, and pause duration in the speech signal. The findings were correlated with the linear structure of interviews as well as question type. Our analysis indicates that the weight of emotion words for the overall expressed emotion is stronger in later interview parts as well as after open questions and meaning questions. Similar patterns were found for pitch and pause duration, but not for vocal effort. Although the verbal expression of emotions was somewhat correlated to pause duration, the hypothesized correlation between the verbal and nonverbal features was not confirmed. The research also shows that the various expressive layers in the interviews as well as the relations between them are a suited basis for computational modeling that may help track emotional personal narratives in interview collections. Additional research is needed to further develop the framework for the automated analysis of verbal and nonverbal cues to automatically generate annotations to be used for exploring spoken word collections.",
keywords = "METIS-307450, IR-93345, EWI-25354",
author = "Truong, {Khiet Phuong} and Westerhof, {Gerben Johan} and S.M.A. Lamers and {de Jong}, {Franciska M.G.}",
year = "2014",
month = "12",
doi = "10.1093/llc/fqu041",
language = "Undefined",
volume = "29",
pages = "621--636",
journal = "Literary and linguistic computing",
issn = "0268-1145",
publisher = "Oxford University Press",
number = "4",

}

TY - JOUR

T1 - Towards modeling expressed emotions in oral history interviews: Using verbal and nonverbal signals to track personal narratives

AU - Truong, Khiet Phuong

AU - Westerhof, Gerben Johan

AU - Lamers, S.M.A.

AU - de Jong, Franciska M.G.

PY - 2014/12

Y1 - 2014/12

N2 - The article aims to model the verbal and prosodic features of emotional expression in interviews to investigate the potential for synergy between scholarly fields that have the narrative as object of study. Using a digital collection of oral history interviews that contains narrative aspects addressing war and violence in Croatia, we analyzed emotional expression through the words spoken, and through the pitch, vocal effort, and pause duration in the speech signal. The findings were correlated with the linear structure of interviews as well as question type. Our analysis indicates that the weight of emotion words for the overall expressed emotion is stronger in later interview parts as well as after open questions and meaning questions. Similar patterns were found for pitch and pause duration, but not for vocal effort. Although the verbal expression of emotions was somewhat correlated to pause duration, the hypothesized correlation between the verbal and nonverbal features was not confirmed. The research also shows that the various expressive layers in the interviews as well as the relations between them are a suited basis for computational modeling that may help track emotional personal narratives in interview collections. Additional research is needed to further develop the framework for the automated analysis of verbal and nonverbal cues to automatically generate annotations to be used for exploring spoken word collections.

AB - The article aims to model the verbal and prosodic features of emotional expression in interviews to investigate the potential for synergy between scholarly fields that have the narrative as object of study. Using a digital collection of oral history interviews that contains narrative aspects addressing war and violence in Croatia, we analyzed emotional expression through the words spoken, and through the pitch, vocal effort, and pause duration in the speech signal. The findings were correlated with the linear structure of interviews as well as question type. Our analysis indicates that the weight of emotion words for the overall expressed emotion is stronger in later interview parts as well as after open questions and meaning questions. Similar patterns were found for pitch and pause duration, but not for vocal effort. Although the verbal expression of emotions was somewhat correlated to pause duration, the hypothesized correlation between the verbal and nonverbal features was not confirmed. The research also shows that the various expressive layers in the interviews as well as the relations between them are a suited basis for computational modeling that may help track emotional personal narratives in interview collections. Additional research is needed to further develop the framework for the automated analysis of verbal and nonverbal cues to automatically generate annotations to be used for exploring spoken word collections.

KW - METIS-307450

KW - IR-93345

KW - EWI-25354

U2 - 10.1093/llc/fqu041

DO - 10.1093/llc/fqu041

M3 - Article

VL - 29

SP - 621

EP - 636

JO - Literary and linguistic computing

JF - Literary and linguistic computing

SN - 0268-1145

IS - 4

ER -