DiffEye: Diffusion-Based Continuous Eye-Tracking Data Generation Conditioned on Natural Images

We sincerely thank the reviewer for their detailed assessment and insightful questions. We are glad they recognize the potential significance and novelty of our diffusion-based approach for modeling human gaze. Below, we address the weaknesses and questions raised in the review.

---

I think the biggest weakness of the paper is that it's not clear to me what it is about modelling the gaze trace data that seems to help (see Questions section below). In addition, while the model is motivated by the need to account for individual differences in gaze behavior, I see no evidence that this model currently actually does this. The data is described in general as collected from different subjects, but then as far as I can tell the model is only conditioned on the stimulus image, not the subject. Assuming I have not misunderstood this, then the individual differences aspect must be clearly described as a possible future extension, rather than something the model currently allows.

The reviewer correctly points out that our model is conditioned on the stimulus image and not on a specific subject ID. We apologize for any ambiguity in our initial framing. Our primary motivation was not to model specific individuals, but rather to capture the inherent variability observed across different human subjects when viewing the same image, and we will clarify this in the final version of the paper

Traditional deterministic models predict a single "average" scanpath, which fails to represent the stochastic nature of scanpaths. Our generative approach, by training on raw trajectories from multiple subjects, learns a distribution of plausible gaze behaviors for a given image. The goal is to generate samples that reflect the diversity seen in the human population, not to replicate a particular person's gaze pattern. As the reviewer suggests, conditioning on subject-specific information is an interesting and important direction for future research, building upon the foundation we establish here for modeling population-level gaze dynamics.

---

Question 1: Why does modeling raw gaze traces work better?

This is an excellent question. The reviewer rightly intuits that the benefit goes beyond simply having "more data." Our hypothesis, supported by our results, is that raw eye-tracking trajectories contain a wealth of spatio-temporal information that is discarded when input data only is scanpaths. For instance, in the MIT1003 dataset, the average raw trajectory has about 724 timesteps, while the average scanpath has only 8.4. This lost information includes:

• Saccadic Dynamics: The precise curvature, velocity, and acceleration profiles of saccades are lost in scanpath representations. Our model can learn these subtle but important characteristics from the continuous trajectory data.
• Suitability for Diffusion Models: Diffusion models excel at learning complex distributions over continuous, high-dimensional data. Raw gaze trajectories, which are essentially smooth time-series data, are a natural fit for this modeling paradigm. In contrast, scanpaths`are discrete, lower-dimensional sequences, and modeling them directly with diffusion is less straightforward.

The reviewer's suggestion to train on linearly interpolated trajectories is a very insightful way to perform an ablation study. Such an experiment would allow us to precisely disentangle the contributions of saccade curvature and timing. It is a fantastic idea for a follow-up study. Our paper's primary contribution is to demonstrate that using a diffusion framework on the full, raw trajectory data is not only feasible but also outperforms methods that rely on only scanpaths, with the caveat that our experimental findings are limited to datasets containing the raw eye tracking data (a limitation we will make clear in the final paper)..

---

Question 2: Choice of 10 fixations for other models.

We thank the reviewer for this important question. The decision to set the number of fixations to 10 for models such as DeepGaze III and IOR-ROI was not an arbitrary choice, but rather a constraint imposed by the models themselves. These specific baselines are designed to generate scanpaths of a fixed, predetermined length and cannot produce variable-length outputs.

While we acknowledge that evaluation metrics can be sensitive to scanpath length, our empirical tests showed that the performance of these baselines did not change significantly when this parameter was varied. We therefore adopted a length of 10 for consistency across these models.

This constraint highlights a key advantage of our method. DiffEye generates a continuous, fixed-duration trajectory (720 timesteps), from which fixations are extracted via post-processing. Consequently, the number of fixations is an emergent property of the generated trajectory, allowing it to vary naturally for each sample, even for the same image. This behavior more closely mirrors that of human observers, whose scanpath lengths are not predetermined.

Number of Fixations

Results are showing the average number of fixations in the generated/ground truth scanpaths for both datasets. As the results show, our model's ability to generate variable-length scanpaths allows it to better match the statistical properties of human gaze across different datasets. For the MIT1003 dataset, the average number of fixations in the ground truth is 8.40, while our model produces scanpaths with 8.48 fixations. Similarly, for the OSIE dataset, our model generates 8.08 fixations per scanpath, closely approximating the ground truth of 9.40. This consistent alignment with human data demonstrates a clear advantage over models that cannot produce variable-length scanpaths.

---

Question 3: Predictions seem too concentrated.

We thank the reviewer for this keen qualitative observation. It is true that in some instances, as shown in Figure 3, the model's generated scanpaths can be more concentrated than the ground truth distribution.

This observation touches on the known challenge of evaluating generative models, where no single metric tells the whole story. The metrics we used—Levenshtein Distance, Discrete Fréchet Distance , DTW, and TDE—are standard in the literature and capture different aspects of sequence and temporal similarity. While our model performs very well on these metrics on average, they may not perfectly capture the "overall spread" or diversity of fixations across all salient objects.

This is precisely why we included extensive qualitative results, to provide a more holistic view of the model's performance. The reviewer's point is well-taken, and developing more robust metrics that specifically evaluate the spatial distribution and spread of generated fixations against the ground truth is a valuable avenue for future work.

---

We apologize for any ambiguity in our initial framing. Our primary motivation was not to model specific individuals, but rather to capture the inherent variability observed across different human subjects when viewing the same image, and we will clarify this in the final version of the paper. Traditional deterministic models predict a single "average" scanpath, which fails to represent the stochastic nature of scanpaths. Our generative approach, by training on raw trajectories from multiple subjects, learns a distribution of plausible gaze behaviors for a given image. The goal is to generate samples that reflect the diversity seen in the human population, not to replicate a particular person's gaze pattern.

Unfortunately this response makes me question the authors' appropriate understanding of this literature. First, "individual differences" has a specific meaning, which is exactly conditioning on subject-specific information. That the authors conflate this with stochastic gaze behavior is disappointing. Second, the writing in the reply implies that this model is the first to represent the stochastic nature of scanpaths. This is not true: for example, the DeepGaze family of models (e.g. [1], [2]) are explicitly probabilistic, which means that they capture stochasticity in a principled way. Please clarify this appropriately in any revision.

I think this paper would be stronger had the authors been able to include at least one a linear trajectory control condition to answer the "why" question. But I understand that this might not have been feasible in the remaining time.

Given the points above, I would keep my score as is. However, I find the authors' responses to the other reviewers to be quite convincing, especially the inclusion of gaze statistics and the new sequence scores in response to 6vm6. (Note however that I suspect there might be something off with the distribution summaries for the direction and angle, which are circular variables). Therefore, I raise my score to 5.

Finally, I'd like to point out regarding questions and responses to some of the other reviewers, that in general, video-based eyetracking data will not have sufficient spatiotemporal resolution to reliably measure microsaccades. Microsaccades are typically defined as rapid eye movements of less than 30 arcmin (0.5 deg; see e.g. [3]). Typically, even the top-range research eyetrackers don't have accuracies reliably < 1 deg. So in my view, none of the published free viewing datasets contain sufficiently sampled data to allow learning of microsaccades, much less other fixational eye movement properties such as ocular drift.

[1] Kümmerer, M., Bethge, M., & Wallis, T. S. A. (2022). DeepGaze III: Modeling free-viewing human scanpaths with deep learning.

[2] Kümmerer, M., Wallis, T. S. A., Gatys, L. A., & Bethge, M. (2017, October). Understanding Low- and High-Level Contributions to Fixation Prediction.

[3] Rucci, M., & Poletti, M. (2015). Control and Functions of Fixational Eye Movements.

We sincerely thank the reviewer for their feedback. Below, we address each of the weaknesses and questions.

---

W1

We acknowledge that all datasets contain biases, including the well-known center bias in many visual attention datasets. However, our model is designed to mitigate this specific issue. Because DiffEye is conditioned on patch-level image features, it learns to associate gaze with specific semantic content within the image, rather than simply memorizing a spatial prior like a center bias.

Our strong performance (see Table 3 in the original paper) on the entirely unseen OSIE dataset serves as direct evidence of this capability. This result demonstrates that our model can generalize beyond the specific characteristics and any potential biases of the MIT1003 training set. We will add a discussion to the limitations section.

---

W2 & Q3 & Q4

Our core hypothesis is that while fixations are where visual information is primarily processed, the full trajectory—which includes saccade velocity, curvature, and microsaccades—contains a richer, continuous spatio-temporal signal. Modeling this complete signal allows for a more faithful representation of the underlying dynamics of visual exploration, which might be useful for certain studies, and there is no cost for this flexibility as we can always extract the corresponding scanpath if needed.

Diffusion models are well-suited for modeling such continuous, high-dimensional data. The smooth nature of raw trajectories is a better representational match for these models than discrete, variable-length scanpaths. This allows DiffEye to learn the nuanced patterns of human eye movements, which is reflected in our strong performance on metrics that explicitly evaluate temporal sequence similarity, like DTW and TDE. We have extracted additional statistics from the eye tracking trajectories based on the reviewer’s suggestion. While we can’t include a figure in the rebuttal, please find the statistics in the tables below:

Saccade Amplitude (x10² pixels)

Saccade Direction (degrees)

Inter-Saccade Angle (degrees)

Number of Fixations

While we cannot put the statistics for OSIE dataset due to the character limitation, and can't upload figure, we will put them to the final version. This analysis demonstrates that our approach generates scanpaths whose statistical properties closely mirror the ground truth.

---

W3 & W4 & Q1 & Q2 & Q5 & Q6

We will significantly expand our methods section to provide the following clarifications.

• Diffusion Process Details:

  • Forward Process: For each training sample, a diffusion timestep $t_{diff}$ is uniformly sampled from {1, ..., T_diff}, where $T_{diff} = 1000$. At $t=0$, we have the original, noise-free trajectory $R^{(0)}$. Noise is added to $R^{(0)}$ according to a pre-defined linear noise schedule with variance ranging from $1 \times 10^{-4}$ to $2 \times 10^{-2}$, producing the noised trajectory $R^{(t_{diff})}$.
  • Reverse Process: The U-Net, $\epsilon_\theta$, is trained to predict the total noise $\epsilon$ that was added to the original trajectory $R^{(0)}$ to obtain $R^{(t_{diff})}$. For inference, standard diffusion models (DDPMs) can be very slow, as they require reversing the process through all 1000 steps. To improve efficiency, we use a DDIM [66] scheduler. DDIM introduces a more flexible, non-Markovian diffusion process that allows the model to take larger "jumps" during the reverse denoising phase. Instead of taking 1000 small steps, DDIM enables high-quality sample generation in significantly fewer steps. This allows us to approximate the reverse process in just 50 steps at inference time, greatly speeding up generation without a major compromise in quality.

• Stimulus Conditioning and CPE Details:

  • Conditioning Mechanism: We use feature maps, not class tokens, for conditioning. FeatUp provides a high-resolution feature map for a given image. We interpolate this map to a $32 \times 32$ grid, resulting in 1024 patch tokens. These patch tokens serve as the key and value inputs to cross-attention layers located within each block of our U-Net. The trajectory tokens (augmented with positional information) serve as the query.
  • Backpropagation and CPE: The loss function is the L2-norm between the ground-truth noise $\epsilon$ and the U-Net's predicted noise $\epsilon_\theta$. Our proposed CPE module is a simple, parameter-free addition operation. Because it is just an addition, gradients flow directly through it to the layers that precede it. This end-to-end process forces the U-Net to learn the correspondence between the visual content (from the patch features) and the gaze location (from the trajectory) to successfully predict and remove the noise. The backpropagated gradients update all trainable parameters, including the U-Net weights, the cross-attention modules, and the initial linear layers that project the raw inputs into the shared embedding dimension.

---

W5

We will clarify this phrasing. The vast majority of trajectories in the dataset were shorter than our target length and were padded. A small fraction of sequences that were slightly longer than 720 timesteps were minimally downsampled to fit the fixed-length requirement of the model architecture. Most importantly, the core temporal structure was preserved. We will make it clear that most trajectories were truncated or padded, and only a few were lightly downsampled, with a negligible effect on the signal.

---

W6

Our intent in the main paper (e.g., Figure 3) was to illustrate that the entire set of our generated trajectories successfully captures the overall distribution of the ground-truth trajectories from all 15 observers. This demonstrates our model’s ability to learn population-level variability, not just replicate a single average path.

However, your point about showing a direct one-to-one similarity is crucial. Quantitatively, the"Best" score in Tables 2 and 3 is designed to capture exactly this: it reflects the average distance between each ground-truth scanpath and its single closest match among all generated scanpaths. To make this more intuitive, we will add a new qualitative figure to the appendix showing these one-to-one "best match" comparisons to visually demonstrate our model's ability to replicate specific human gaze patterns.

---

W7

The metrics were chosen to fit our primary task of generating dynamic sequences, which differs fundamentally from static saliency prediction. Metrics like the AUC are excellent for evaluating static saliency maps, but they are agnostic to temporal order, path shape, and the dynamics of how a scene is explored. They cannot distinguish between two different scanpaths that happen to cover the same salient regions in a different order.

In contrast, metrics like DTW and Levenshtein Distance are specifically designed to measure the similarity between two entire sequences.

• DTW is particularly valuable as it aligns sequences that may vary in speed, capturing similarity in the overall shape and flow of a path.
• Levenshtein Distance measures the "edit distance" between two fixation sequences, providing a robust measure of similarity in both the spatial locations and the order of fixations.

By using these sequence-based metrics, we perform a "distribution-wise" evaluation. A low average score signifies that the distribution of paths produced by our model is structurally and temporally similar to the human distribution. While we evaluate the secondary task of saliency prediction using standard metrics like AUC (provided in the supplementary material), our primary contribution is best assessed by these sequence-aware metrics.

We provide results on Sequence Score (SS) and Semantic Sequence Score (SemSS) on MIT1003, which further validate our choice of sequence-based evaluation, demonstrating our model's superior ability to capture the spatio-temporal dynamics of human gaze.

We sincerely thank the reviewer for their detailed and constructive feedback. We appreciate the acknowledgment of our work's novelty in its hypothesis (S1) and methodology (S2). Below, we address the identified weaknesses and questions by quoting the specific concerns and providing our clarifications.

---

W1: Value Proposition of Raw Trajectories

"The claim by the paper than scanpath is a 'compression of trajectories' is flawed... it is not clear what is the actual value proposition of raw trajectories prediction only (without fixations/saliency)."

We used "compressed" in the sense that scanpaths, while capturing the most critical information for cognitive processing, are a lower-dimensional summary of the full spatio-temporal eye movement data. In the final version of the paper we will use more appropriate language.

Our motivation for modeling raw trajectories is twofold. First, it provides a more fundamental and flexible representation of the gaze data produced by eye tracking. By generating the foundational raw signal, our approach allows end-users the "flexibility to define and extract fixations using any desired approach". This is beneficial because different eye tracking studies and products might use different approaches to extract the scanpath. Our model provides the underlying data, making it adaptable to any specific post-processing pipeline. From the modeling perspective, our approach avoids the need for arbitrary decisions that have plagued prior scanpath-based methods. For example, DeepGaze III and IOR-ROI require the number of fixations to be specified as a fixed input for scanpath synthesis, and this may not be straightforward to specify for a given stimuli. In contrast, DiffEye generates a continuous trajectory. Consequently, the number of fixations emerges naturally as a property of the generated data. This allows the model to produce scanpaths with an adaptive number of fixations for each image, which better reflects the natural diversity of human viewing patterns. As shown in the table below, mean number of fixations generated by our model is closer to the original statistics.

Number of Fixations

Second, raw trajectories provide a richer signal for generative models. They contain fine-grained spatio-temporal information, like microsaccades and saccadic curvature, which is lost during scanpath extraction. As noted in our paper, raw trajectories in the MIT1003 dataset average 723.7 timesteps, while scanpaths average only 8.4, indicating a "substantial reduction in spatio-temporal information". Diffusion models have proven highly effective at learning complex, high-dimensional data distributions, making them a natural and powerful choice for modeling the intricate patterns found in raw eye-tracking data. By learning from this richer data space, our model can produce more realistic and nuanced gaze behaviors, which is supported by our superior performance on trajectory-based metrics.

---

W2 & W3: Datasets and Evaluation Metrics

"The comparison of the scanpath and saliency prediction results are limited to 2 datasets... The calls into the question of the generalization of the proposed method..." "The evaluation metrics are non-standard for visual attention community... But the paper only uses trajectories-based metrics." We understand the reviewer's concerns regarding the scope of our evaluation. Our choices were guided by our core hypothesis and significant practical data constraints.

On Datasets: Our method's primary requirement is the availability of raw eye-movement trajectory data, not just scanpaths. The MIT1003 dataset is "the only publicly available dataset that provides raw eye-tracking data for natural images obtained during a free-viewing task". While datasets like COCO-FreeView are larger, they only contain scanpath data and thus cannot be used to train our model, which operates on the raw signal. Our purpose in this work is not to claim state-of-the-art scanpath prediction across all datasets, but to introduce and validate a novel methodology for generating continuous eye-tracking data, a task no prior work has addressed for natural images. We demonstrated strong generalization by evaluating on "the entirely unseen OSIE dataset", where our model performed competitively without any fine-tuning. We will modify the final version of the paper to clarify our claim for performance.

On Evaluation Metrics For Scanpaths: We chose trajectory-based metrics (Levenshtein, DTW, DFD) because our model is inherently generative, and these metrics excel at comparing the similarity between entire sequences, capturing both spatial and temporal structures. This aligns with our goal of evaluating the model's ability to replicate the distribution of human gaze behavior. However, based on the reviewe’s comment we added additional SS and SemSS metrics. Please find SS and SemSS results below:

MIT1003 Results

OSIE Results

Our model achieves state-of-the-art performance on the MIT1003 dataset, excelling in both Sequence Score and Semantic Sequence Score. Furthermore, the model generalizes effectively by remaining competitive on the unseen OSIE dataset, validating that the rich signal from raw trajectories produces high-quality scanpaths. This strong generalization stands in sharp contrast to models like ROI, which, having been trained on the OSIE dataset, performs well there but fails to produce competitive results on the MIT dataset. Notably, our model’s robust performance was achieved with significantly less data than competing models. While DeepGazeIII was trained on approximately 600,000 scanpaths from 11,000 images, our model used only 8,900 trajectories from 1,000 images. We attribute any minor performance differences to this vast disparity in data scale, demonstrating that by using rich trajectory information, our method can achieve competitive, and even superior, results with less than 10% of the training images.

On Evaluation Metrics For Saliency: We would like to clarify that we did perform a full evaluation using standard saliency metrics (AUC-Judd, AUC-Borji, NSS, SIM, CC, KL). These results are presented in Table 4 in the supplementary material.

---

W4: Modeling Diversity

"(minor) Paper claims that the current models are unable to capture the diversity of individual viewers. But there is no attempt to model individual differences in their approach. Hence, this point, while true, is irrelevant to their work."

We thank the reviewer for this clarifying comment.

Our point was simply that stochastic generative models are advantageous relative to deterministic synthesis approaches that produce a single, "average" scanpath for a given image, which fails to capture the natural diversity in gaze trajectories arising when different subjects attend to different regions.

Our diffusion-based approach inherently addresses the stochasticity of eye tracking data across subjects. By learning the entire distribution of trajectories from the population data, our model can be sampled to generate a diverse set of plausible outputs for a single input image. Each generated sample is a trajectory that could have been produced by a human viewer, thereby capturing the variability observed across the population. We will revise the phrasing in the abstract and introduction to make this important distinction clear, and mention the characterization of individual differences as a topic for future work

Dear Reviewer ftE1,

Thank you for your thoughtful and constructive review of our manuscript. We appreciate your valuable feedback.

We are encouraged that you recognized the core strengths of our work, including:

• The clarity of the manuscript and the effectiveness of the figures.
• The value of our approach in training directly on raw eye-movement trajectories.
• The appropriate use of a probabilistic model to account for the inherent variability in human gaze.

---

I would be critical about the claim that the system helps “to better understand how the complex human attention system operates on images”. This is one big data fitting exercise. Scientifically, we do not understand the human attention system better.

We agree with this statement, and will remove this sentence from the final paper and focus on positioning our work in the context of human gaze trajectory generation.

---

Can the authors give the reader an intuitive explanation for the necessity of the CPE alignment beyond the statement: "Corresponding Positional Embedding (CPE) To further improve spatial alignment, we introduce a novel positional embedding method, Corresponding Positional Embedding (CPE)"?

Thanks for this useful question, we provide an in-depth explanation below, and we will add this discussion to the final version of the paper. The core challenge in generating a realistic eye-tracking trajectory is to teach the model not just what is in an image, but precisely where the eye should look in relation to that content. Our initial attempts to condition the model using a single global image feature vector led to poor generation quality (see Table 2 w/o Patch Level Features), as the global feature lacks localized spatial semantics necessary for effective conditioning. The model understood the image's global context but could not ground the gaze path in specific local regions. CPE solves this by providing a common spatial "address system" for both the visual input and the gaze trajectory.

1. Creating the "Map": We first construct a 2D positional embedding grid, denoted as $P \in \mathbb{R}^{H \times W \times D}$, which acts as a universal map based on the original image's resolution ($H \times W$). This grid assigns a unique positional vector—an "address"—to every coordinate in the image space.

2. Tagging the Trajectory: For each coordinate $(x_i, y_i)$ in the eye-tracking trajectory $R$ , we retrieve its corresponding positional address $p_i$ from the map $P$ and add it directly to that trajectory point's embedding. The final augmented trajectory token is computed as: $R_{i}^{CPE} = R_{proj}[i] + p_{i}$. The trajectory is now a sequence of coordinates, each tagged with its absolute position on the shared map.

3. Tagging the Image Content: Crucially, we use the exact same map $P$ for the image's patch-level features, $F_{proj}$. We interpolate the map $P$ to the patch resolution, yielding $P' \in \mathbb{R}^{H' \times W' \times D}$, and add these positional addresses to their corresponding image patch features. This is computed as: $F_{CPE} = F_{proj} + P'$. Now, each piece of visual content is also tagged with its absolute position on the same shared map.

"By sharing and aligning positional information across both trajectories and image patches, CPE enables effective spatial correspondence and interaction between gaze behavior and visual content". Because both the gaze location and the image content now speak the same spatial language, our model's cross-attention mechanism can effectively associate a gaze point with the visual content at that exact location.

The importance of this component is empirically demonstrated in our ablation study (Table 2). When CPE is removed (w/o CPE), the model's performance degrades significantly across all metrics, especially those sensitive to spatial alignment like Levenshtein and Fréchet distances. This confirms that CPE's explicit spatial alignment is a critical contribution that enhances the model's ability to effectively localize attention signals.

---