Using sensors to monitor and predict momentary feelings: Exploring how multimodal passive sensing of sleep, physical activity, mobility and smartphone use supports within-person predictions of momentary feelings in daily life

Mental health concerns such as stress-related complaints, depression, and burnout often develop gradually and fluctuate within individuals over time. Yet, in practice, symptoms are often only detected after they have escalated, partly because assessments commonly rely on sporadic retrospective questionnaires that provide a static “snapshot” and are vulnerable to recall bias. The growing availability of smartphones and wearable sensor technologies (e.g., smart rings) offers an opportunity to complement self-reports by continuously and passively capturing behavioral and physiological patterns in naturalistic settings, such as sleep, physical activity, mobility, and smartphone usage. Furthermore, smartphones can also provide a promising alternative to retrospective questionnaires, namely Ecological Momentary Assessments (EMA). EMA consists of short surveys or questions that can be sent to the user via smartphones several times a day to capture fine-grained momentary feelings in daily life, close to their occurrence. However, such high-frequency self-reports can be burdensome for participants and difficult to sustain. Currently, it is still not well understood how these multimodal passive sensing signals (i.e. data from smartphones and wearables) relate to momentary feelings (measured through EMA) over longer time periods, and how much these relations differ across individuals. It also remains unclear how such data can be used for personalized monitoring and prediction in both preventive and restorative contexts. The central aim of this dissertation is to determine how patterns of sleep, physical activity, mobility, and smartphone usage can be used to monitor and predict momentary feelings and mental health over time. These patterns were derived from multimodal devices (smartphone, smart ring), combining continuous passive sensing with daily EMA. A key focus is on within-person processes: how these relationships evolve for each individual, and how they can support personalized mental health monitoring. Two longitudinal cohorts were studied with a multimodal measurement setup: (1) a preventive cohort of 16 PhD candidates monitored for nine months, and (2) a restorative cohort of 18 employees returning to work after stress- or burnout-related sick leave monitored for six months. In each cohort, continuous passive sensing data (smart ring and smartphone) were combined with daily EMA of momentary feelings and repeated retrospective questionnaires. In the preventive cohort, EMA was used to assess daily valence (how pleasant a person felt) and arousal (how mentally active or passive a person felt), whereas in the restorative cohort, EMA was used to assess daily emotional exhaustion (how depleted a person felt emotionally). The dissertation began with descriptive and correlational analyses linking passive sensing signals to momentary feelings and then progressed to individual-level (subject-dependent) predictions of momentary feelings from passive sensing signals using machine learning algorithms (Random Forests). In the final part of the dissertation, it was examined how daily EMA-based aggregates of emotional exhaustion relate to monthly retrospectively measured burnout symptoms. Across the empirical chapters and two cohorts, six central patterns emerged. First, sleep and physical activity were relatively robust indicators, with sleep timing, sleep variability across individuals, and activity-related features (including sedentary time and activity-related metrics) frequently contributing to explaining day-to-day variation in momentary feelings, with particularly pronounced relevance in the restorative cohort during recovery and reintegration. Second, mobility and smartphone usage exhibited more heterogeneous, context-dependent patterns, with these features being sometimes informative in individual-level (subject-dependent, i.e. models tailored to each subject) models but less consistent across individuals. Third, inter-individual differences in associations with momentary feelings were substantial across all domains, indicating that the same sensor-derived signal can vary substantially across individuals and contexts. Fourthly, individual-level (subject-dependent) modeling approaches significantly outperformed group-level (subject-independent, i.e. models based on group patterns) approaches when predicting momentary feelings with multimodal passive sensing data. Fifthly, participants’ a priori beliefs about behavioral and physiological patterns often aligned with empirical feature importance, suggesting that user insights might help guide feature selection and data collection rather than relying exclusively on fully data-driven approaches. Sixth, comparisons between daily EMA and retrospective questionnaires showed that these measures capture different temporal aspects of burnout, with EMA being particularly sensitive to within-person changes, highlighting their suitability for personalized monitoring approaches. Taken together, these findings illustrate the need for personalized, within-person approaches. By integrating multimodal passive sensing with intensive longitudinal EMA across both preventive and restorative contexts, this dissertation advances towards a more dynamic, person-centered perspective on mental health monitoring. The findings suggest that EMA plays a crucial role in validating and interpreting passive sensing signals. This also points toward a longer-term perspective in which, once models are sufficiently personalized and validated, passive sensing could reduce reliance on frequent EMA, thereby lowering participant burden while preserving the assessment of subjective experience. Practical implications include the potential development of personalized monitoring systems for high-risk or vulnerable groups (e.g., individuals recovering from burnout), which could use passive sensing data to detect early warning signs and deliver timely, tailored support through Just-In-Time Adaptive Interventions (JITAIs). Future research should further evaluate hybrid prediction modeling strategies that combine group-level information with personalized modeling over time. When personal data is still scarce, models may initially rely partly on group-level patterns (e.g., sleep or activity rhythms), but gradually become more personalized as more individual data becomes available. However, highly context-dependent
domains such as mobility and smartphone routines may require personalized modeling from the outset.