A TDRL Model for the Emotion of Regret

Joost Broekens, Laduona Dai

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

Abstract

To better understand the nature, function and elicitation conditions of emotion it is important to approach studying emotion from a multidisciplinary perspective involving psychology, neuroscience and affective computing. Recently, the TDRL Theory of Emotion has been proposed. It defines emotions as variations of temporal difference assessments in reinforcement learning. In this paper we present new evidence for this theory. We show that regret-a negative emotion that signifies that an alternative action should have been taken given new outcome evidence-is modelled by a particular form of TD error assessment. In our model regret is attributed to each action in the state-action trace of an agent for which-after new reward evidence-an alternative action becomes the best action in that state (the new argmax) after adjusting the action value of the chosen action in that state. Regret intensity is modeled as the difference between this new best action and the adjusted old best action, reflecting the additional amount of return that could have been received should that alternative have been chose. We show in simulation experiments how regret varies depending on the amount of adjustment as well as the adjustment mechanism, i.e. Q-trace, Sarsa-trace, and Monte Carlo (MC) re-evaluation of action values. Our work shows plausible regret attribution to actions, when this model of regret is coupled with MC action value update. This is important evidence that regret can be seen as a particular variation of TD error assessment involving counterfactual thinking.

Original languageEnglish
Title of host publication2019 8th International Conference on Affective Computing and Intelligent Interaction, ACII 2019
PublisherIEEE
ISBN (Electronic)9781728138886
DOIs
Publication statusPublished - Sep 2019
Event8th International Conference on Affective Computing and Intelligent Interaction, ACII 2019 - Cambridge, United Kingdom
Duration: 3 Sep 20196 Sep 2019
Conference number: 8
http://acii-conf.org/2019/

Conference

Conference8th International Conference on Affective Computing and Intelligent Interaction, ACII 2019
Abbreviated titleACII
CountryUnited Kingdom
CityCambridge
Period3/09/196/09/19
Internet address

Keywords

  • Computational Modelling
  • Emotion
  • Regret
  • TDRL Emotion Theory

Fingerprint

Dive into the research topics of 'A TDRL Model for the Emotion of Regret'. Together they form a unique fingerprint.

Cite this