ACM IMWUT · Vol. 9, No. 3, Article 143 · September 2025

Through the Eyes of Emotion: A Multi-faceted Eye Tracking Dataset for Emotion Recognition in Virtual Reality

Tongyun Yang†1, Bishwas Regmi†1, Lingyu Du1, Andreas Bulling2, Xucong Zhang1, Guohao Lan1

1Delft University of Technology  ·  2University of Stuttgart  ·  †equal contribution

Request dataset access Paper PDF Code & tools BibTeX
Three-panel figure: (a) Pupil Labs eye-tracking add-on inside an HTC VIVE Pro headset, (b) a participant seated wearing the headset during data collection, (c) the participant's field of view inside the virtual environment.
(a) Pupil Labs add-on eye tracker on an HTC VIVE Pro; (b) a participant during data collection; (c) the participant's field of view in the virtual environment.

Abstract

Virtual Reality (VR) is transforming cognitive and psychological research by enabling immersive simulations that elicit authentic emotional responses. The high demand for VR-based emotion recognition is also evident in fields such as mental healthcare, education, and entertainment, where understanding users' emotional states can enhance user experience and system effectiveness. However, the lack of comprehensive datasets hinders progress in VR-based emotion recognition. In this paper, we present a comprehensive, multi-faceted eye-tracking dataset collected from 26 participants using 28 emotional video stimuli rendered in a custom virtual environment. Our dataset is the first to incorporate high-frame-rate periocular videos, capturing subtle motions, such as micro-expressions and eyebrow shifts, which are critical for emotion analysis. Additionally, it includes high-frequency eye-tracking data, offering gaze direction and pupil dynamics at four times the frequency of existing datasets. Our dataset is also unique in providing emotion annotations according to Ekman's emotion model and, as such, offering experiments impossible using existing datasets. Our benchmark evaluations show that fusing the multi-faceted eye-tracking signals in our dataset significantly improves emotion recognition accuracy. As such, our work has the potential to significantly accelerate and enable entirely new research on emotion-aware VR applications.

What's in the dataset

Each of the 26 participants watched 14 emotional film clips (two per emotion, drawn from a 28-clip stimulus pool validated by Zupan et al.) inside a custom Unity VR environment on an HTC VIVE Pro with a Pupil Labs eye-tracking add-on. Every session is recorded across synchronized modalities:

Periocular videobinocular near-eye cameras, 120 fps, 400 × 400 px
Gaze direction2D gaze estimates at 240 Hz
Pupil diametermeasurements at 120 Hz
Scene videoparticipant's field of view in VR
Head motionIMU signals from the headset
Audiomicrophone recordings per session
Emotion labelsEkman's seven basic emotions with segment-based, self-reported intensity ratings
Timing annotationsper-participant timestamps of reported emotion elicitation

The dataset ships in two releases: the raw recordings total over 10 TB, and the processed, training-ready HDF5 release used for the experiments in the paper comes to about 2 TB. Both are accompanied by the open-source Unity recording interface, the annotation toolchain, and reference preprocessing and benchmark code. File-level documentation lives in the GitHub repository.

Example recordings

Field of view with the gaze estimate overlaid on the stimulus, alongside the two periocular recordings with pupil measurements overlaid.

Disgust
Fear
Surprise
Anger

Dataset access

The dataset contains recordings of human participants, so it is not available for direct download. Access is granted to researchers after a short review. Three steps:

  1. Complete the Dataset Access Request Form

    Tell us who you are, your institution, and how you plan to use the data.

    Open the request form

  2. Sign the Data Use and Confidentiality Agreement

    The agreement covers research-only use, no redistribution, and no attempts to identify participants.

    Open the agreement

  3. Receive the download link by email

    We review each request and email the download instructions to your institutional address once your request is approved.

Please use an institutional email address — the download link is sent there after review. The dataset is provided for non-commercial research and educational purposes only, under the terms of the agreement. Note the download sizes when planning storage: over 10 TB for the raw release, about 2 TB for the processed release.

License

The paper is published under a CC BY-NC 4.0 license and the code under the MIT license. The dataset itself is distributed under the terms of the Data Use and Confidentiality Agreement above.

Citation

If the dataset or tools help your research, please cite:

@article{yang2025VREyeEmotion,
  title     = {Through the Eyes of Emotion: A Multi-faceted Eye Tracking Dataset
               for Emotion Recognition in Virtual Reality},
  author    = {Yang, Tongyun and Regmi, Bishwas and Du, Lingyu and
               Bulling, Andreas and Zhang, Xucong and Lan, Guohao},
  journal   = {Proceedings of the ACM on Interactive, Mobile, Wearable and
               Ubiquitous Technologies},
  volume    = {9},
  number    = {3},
  articleno = {143},
  year      = {2025},
  publisher = {Association for Computing Machinery},
  doi       = {10.1145/3749545}
}

Contact

Questions about the dataset, the tools, or an access request:

Acknowledgements

This work was supported in part by the Meta Research Award, SURF Research Cloud grant EINF-6360, and the EU's Horizon Europe HarmonicAI project under the HORIZON MSCA-2022-SE-01 scheme, grant agreement 101131117. The contents of this page do not necessarily reflect the positions or policies of the funding agencies.