Abstract
Background: Artificial intelligence (AI) tools hold much promise for mental healthcare by increasing the scalability and accessibility of care. However, current development and evaluation practices of AI tools limit their meaningfulness for healthcare contexts and thereby, the practical usefulness of such tools for professionals and clients alike.
Objective: To move towards meaningful evaluation of AI tools in eMental health, this article demonstrates the evaluation of an AI monitoring tool that detects the need for more intensive care in an online grief intervention for older mourners who have lost their spouse.
Method: We leverage the insights from three evaluation approaches: (1) the F1-metric evaluates the tool’s capacity to classify user monitoring parameters as (a) in need of more intensive support, or (b) recommendable to continue using the online grief intervention as is; (2) we use linear regression to assess the predictive value of users’ monitoring parameters for clinical changes in grief, depression, and loneliness over the course of a 10-week intervention. Finally, (3) we collect qualitative experience data from eCoaches (N=4) who incorporated the monitoring in their weekly e-mail guidance during the 10-week intervention.
Results: (1) Based on N=174 binary recommendation decisions, the F1-score of the monitoring tool was 0.91. (2) Due to minimal change in depression and loneliness scores after the 10-week intervention, only one linear regression was conducted. The difference score in grief before and after the intervention was included as dependent variable. Participants’ (N=21) mean score on the selfreport monitoring, the estimated slope of individually fitted growth curves, and its standard error (i.e. participants’ response pattern to the monitoring questions) were used as predictors. Only the mean
monitoring score exhibited predictive value for the observed change in grief (R2 =1.19, SE 0.33, t(df) = 3.58(16), P=.002). (3) The eCoaches appreciated the monitoring tool as an opportunity a) to confirm their initial impression about intervention participants, b) for personalizing their e-mail guidance and c) to detect when participants’ mental health deteriorated during the intervention.
Conclusion: The monitoring tool evaluated in this article identifies a need for more intensive support reasonably well in a non-clinical sample of older mourners, has some predictive value for the change in grief symptoms during a 10-week intervention, and is appreciated as an additional source of mental health information by eCoaches who supported mourners during the intervention. Each evaluation approach in this article comes with its own set of limitations, including (a) skewed class distributions in prediction tasks based on real-life health data and (b) choosing meaningful statistical analyses based on clinical trial designs that are not targeted at evaluating AI tools. However, combining multiple evaluation methods facilitates drawing meaningful conclusions about the clinical value of AI monitoring tools for their intended mental health context.
Objective: To move towards meaningful evaluation of AI tools in eMental health, this article demonstrates the evaluation of an AI monitoring tool that detects the need for more intensive care in an online grief intervention for older mourners who have lost their spouse.
Method: We leverage the insights from three evaluation approaches: (1) the F1-metric evaluates the tool’s capacity to classify user monitoring parameters as (a) in need of more intensive support, or (b) recommendable to continue using the online grief intervention as is; (2) we use linear regression to assess the predictive value of users’ monitoring parameters for clinical changes in grief, depression, and loneliness over the course of a 10-week intervention. Finally, (3) we collect qualitative experience data from eCoaches (N=4) who incorporated the monitoring in their weekly e-mail guidance during the 10-week intervention.
Results: (1) Based on N=174 binary recommendation decisions, the F1-score of the monitoring tool was 0.91. (2) Due to minimal change in depression and loneliness scores after the 10-week intervention, only one linear regression was conducted. The difference score in grief before and after the intervention was included as dependent variable. Participants’ (N=21) mean score on the selfreport monitoring, the estimated slope of individually fitted growth curves, and its standard error (i.e. participants’ response pattern to the monitoring questions) were used as predictors. Only the mean
monitoring score exhibited predictive value for the observed change in grief (R2 =1.19, SE 0.33, t(df) = 3.58(16), P=.002). (3) The eCoaches appreciated the monitoring tool as an opportunity a) to confirm their initial impression about intervention participants, b) for personalizing their e-mail guidance and c) to detect when participants’ mental health deteriorated during the intervention.
Conclusion: The monitoring tool evaluated in this article identifies a need for more intensive support reasonably well in a non-clinical sample of older mourners, has some predictive value for the change in grief symptoms during a 10-week intervention, and is appreciated as an additional source of mental health information by eCoaches who supported mourners during the intervention. Each evaluation approach in this article comes with its own set of limitations, including (a) skewed class distributions in prediction tasks based on real-life health data and (b) choosing meaningful statistical analyses based on clinical trial designs that are not targeted at evaluating AI tools. However, combining multiple evaluation methods facilitates drawing meaningful conclusions about the clinical value of AI monitoring tools for their intended mental health context.
| Original language | English |
|---|---|
| Publisher | JMIR Publications |
| Number of pages | 31 |
| DOIs | |
| Publication status | Published - 15 Jun 2024 |