Seeing and Speaking with Culture: How Visitor Profiles Shape Multimodal Interaction in a VR Art Exhibition

Understanding how users engage with socially interactive agents in immersive cultural environments is crucial for designing adaptive and contextually appropriate virtual experiences. This study explores how individual user traits—including familiarity with virtual reality, museum visitation habits, prior knowledge, interest in specific artworks, and interaction preferences—relate to gaze behavior and conversational engagement in a VR art exhibition. Fifty-two participants explored a virtual museum featuring five paintings and interacted with agents designed to share information and facilitate discussion. Data were collected through questionnaires, eye-tracking metrics (e.g., fixation duration, scanpath length), and conversational transcripts (e.g., response length, number of user turns). A multimodal analysis aligned gaze and speech data to examine how visual attention and verbal interactions co-occur in relation to user traits. Results indicate that traits such as prior knowledge, interest, and personalization preferences significantly influence both gaze patterns and conversational behaviors. While some effects varied in strength or consistency, the findings offer valuable insights into how personal characteristics shape user-agent interactions. These insights inform the design of adaptive virtual agents and support broader efforts to model multimodal user bahevior for real-time personalization in immersive environment.