Disrupted self, therapy, and the limits of conversational AI

Conversational agents (CA) are thought to be promising for psychotherapy because they give the impression of being able to engage in conversations with human users. However, given the high risk for therapy patients who are already in a vulnerable situation, there is a need to investigate the extent to which CA are able to contribute to therapy goals and to discuss CA’s limitations, especially in complex cases. In this paper, we understand psychotherapy as a way of dealing with existential situations and position CAs in the context of the therapeutic experience of patients. This experience is determined by the patient’s unique personal context and specific therapy goals. We suggest that psychotherapy is a fundamentally dialogical activity, because it crucially involves work on the self and one’s self-narrative. This brings us to our central question: is it possible for CAs to engage in a productive therapeutic dialogue, given their limitations as epistemic agents? We will discuss several of those limitations, show how these undermine the possibility of engaging in a therapeutic dialogue, and illustrate those limitations through discussions of the cases of grief and abuse.