We thank reviewer 7cjL for the helpful feedback. To summarize all reviewer feedback, we are encouraged that the reviewers find our problem formulation interesting and advantageous (7cjL, LuMp, 23sp) while being methodologically solid and innovative (7cjL, LuMp) with clear writing (7cjL, k89X). Additionally, they find that our comprehensive experimental results demonstrate significant performance improvements (7cjL, LuMp, 23sp) with the curation of two new large-scale benchmarks (LuMp, 23sp). Below, we address the specific questions raised in reviewer 7cjL’s comments.

(7cjL-Q1) Detailed analysis on the latency of our framework

Our framework has two primary sources of latency: (i) extracting the relevant set of points from gaze and SLAM keypoints (performed every 100ms), and (ii) performing model inference every second on a set of points spanning two seconds.

Since the point extraction can be pre-computed and stored at 10 fps for continuous use, we only need to consider the computation time for processing the final observation when calculating inference latency. We measured this in a resource-constrained environment compared to our training setup, using 8 CPU cores and a GPU with 12GB VRAM (e.g., Titan X). In the table below, all operations except model inference are processed on the CPU, with the following execution time:

The first stage, which handles outlier removal, axis-aligned bounding box cropping for keypoints, and selection of points within a certain degree of eccentricity from the gaze, can be processed within 10ms. In the second stage, we identified that the primary bottleneck lies in voxelization rather than in the model itself. This occurs because the large number of keypoints from the previous stage should be voxelized, whereas the model operates efficiently once it receives the 3D voxelized representation.

Consequently, the average inference latency is 71.241ms, yielding a real-time factor of 0.036 when processing 2-second input for multi-second forecasting. This confirms our claim that the framework supports fast processing. However, actual AR/VR environments typically operate with even fewer computational resources, and thus additional optimization techniques such as model quantization or more efficient voxelization could be considered for further performance improvements.

(7cjL-Q2) Suggestions on writing

Thank you for your suggestion on the related work. We will include these papers and fix formatting issues.

We thank reviewer LuMp for the constructive comments. To summarize all reviewer feedback, we are encouraged that the reviewers find our problem formulation interesting and advantageous (7cjL, LuMp, 23sp) while being methodologically solid and innovative (7cjL, LuMp) with clear writing (7cjL, k89X). Additionally, they find that our comprehensive experimental results demonstrate significant performance improvements (7cjL, LuMp, 23sp) with the curation of two new large-scale benchmarks (LuMp, 23sp). Below, we address the specific questions raised in reviewer LuMp's comments.

(LuMp-Q1) Additional experiments on FoVS-EgoExo

Due to space constraints, Table 3 previously reported only partial entries. We further report two additional ablation experiments, three 3D localization baselines, and one 2D gaze anticipation baseline:

Overall, the results show a similar performance distribution to those reported on FoVS-Aria except for the 2D baseline often outperforming EgoChoir, presumably due to increased 3D scene complexity compared to FoVS-Aria. Due to time constraints, additional rows will be incorporated into the draft and the response once they become available.

(LuMp-Q2) Detailed analysis on the latency of our framework

Our framework has two primary sources of latency: (i) extracting the relevant set of points from gaze and SLAM keypoints (performed every 100ms), and (ii) performing model inference every second on a set of points spanning two seconds.

Since the point extraction can be pre-computed and stored at 10 fps for continuous use, we only need to consider the computation time for processing the final observation when calculating inference latency. We measured this in a resource-constrained environment compared to our training setup, using 8 CPU cores and a GPU with 12GB VRAM (e.g., Titan X). In the table below, all operations except model inference are processed on the CPU, with the following execution time:

The first stage, which handles outlier removal, axis-aligned bounding box cropping for keypoints, and selection of points within a certain degree of eccentricity from the gaze, can be processed within 10ms. In the second stage, we identified that the primary bottleneck lies in voxelization rather than in the model itself. This occurs because the large number of keypoints from the previous stage should be voxelized, whereas the model operates efficiently once it receives the 3D voxelized representation.

Consequently, the average inference latency is 71.241ms, yielding a real-time factor of 0.036 when processing 2-second input for multi-second forecasting. This confirms our claim that the framework supports fast processing. However, actual AR/VR environments typically operate with even fewer computational resources, and thus additional optimization techniques such as model quantization or more efficient voxelization could be considered for further performance improvements.

(LuMp-Q3) Practical applications of 3D visual span forecasting

As our title "Gaze Beyond the Frame" implies, our 3D representation offers significant competitive advantages over 2D methods in the scenarios involving perception and interaction beyond the current visual frame. While 2D approaches are constrained to forecasting on existing visual frames, our method can predict attention patterns in previously unseen spatial regions. Foveated rendering in VR/MR systems exemplifies this advantage: when users turn their head to attend to different contexts, our 3D spatial awareness maintains consistent predictions across head movements and viewpoint changes, whereas 2D methods struggle with such dynamic scenarios.

Moreover, our approach potentially enables various forms of proactive environmental assistance by integrating ambient intelligence. For instance, systems can adjust lighting or display configurations in spaces where users are predicted to look, facilitating faster information perception. Additionally, by considering users' established focusing patterns, ambient assistants can proactively remind users of information they might otherwise overlook, creating more intuitive human-environment interactions.

Temporally persistent 3D representations offer another key advantage. While 2D-based forecasting becomes less reliable as self-motion increases, our framework utilizes 3D spatial representation that is relatively robust to self-motion. This opens up the potential for extending our framework for forecasting on a longer horizon, which could enable wearable devices to learn and anticipate user behaviors in their daily 3D environments rather than being limited to momentary 2D observations.

Lastly, our experimental results demonstrate that 3D visual span predictions, when back-projected to 2D image planes, achieve performance comparable to 2D-specific models without requiring 2D-specific training. This suggests that our 3D approach not only unlocks new capabilities but also maintains the effectiveness of traditional 2D methods while providing the additional spatial understanding necessary for future egocentric applications.

(LuMp-Q4) Rationale behind our primary evaluation metrics

We primarily build upon the recent 2D egocentric gaze estimation and anticipation literature [17,33,34], where F1, Precision, and Recall are well-established metric choices. We further complement these metrics by incorporating measures that reflect the nature of 3D forecasting: IoU for measuring overlaps between volumetric regions and metric distance (Table 2) to compensate for narrow regions from the foveal span.

To provide complementary analysis on saliency metrics in our benchmarks, we report CC and AUC over the logit distribution of 3D grids using three variants of our framework. Note that AUC is reported instead of KLD due to zero-value sensitivity of KLD [TPAMI’19] observed in our evaluation:

The results demonstrate patterns that are generally consistent with previously reported metrics. We will include this additional analysis in the Appendix.

(LuMp-Q5) Elaboration on users with cognitive and perceptual impairments

Several user groups with cognitive and perceptual impairments could potentially benefit from this technology. These include individuals with visual perceptual deficits who may miss objects in their peripheral vision, those with age-related visual degeneration who experience slower or less accurate visual targeting compared to typical users, and people with mild cognitive impairment who could benefit from contextual reminders about their behavior sequences and task progression.

The implementation of assistive support for these users would vary significantly depending on the underlying device or platform. For instance, in display-equipped systems, highlighted overlays could be used to draw attention to forecasted visual spans, helping users identify areas of upcoming interaction. In environments with voice assistants or ambient intelligence systems, direct or indirect cues (e.g., haptic or voice assistant notification) could be provided to users regarding perceptually overlooked contexts, thereby facilitating both perception and interaction. Such proactive assistance has the potential to reduce cognitive load while maintaining user autonomy by anticipating user needs based on their natural gaze patterns and environmental context.

(LuMp-Q6) Suggestions on future work and relevant papers

Thank you for your suggestion on the related work and future direction. Enhancing subject-aware personalization of 3D forecasting could be an interesting extension by incorporating methods like test-time adaptation. We will discuss them in our final draft.

Reference

[TPAMI’19] Bylinskii et al. (2019). What Do Different Evaluation Metrics Tell Us about Saliency Models?

We thank reviewer 23sp for the constructive feedback. To summarize all reviewer feedback, we are encouraged that the reviewers find our problem formulation interesting and advantageous (7cjL, LuMp, 23sp) while being methodologically solid and innovative (7cjL, LuMp) with clear writing (7cjL, k89X). Additionally, they find that our comprehensive experimental results demonstrate significant performance improvements (7cjL, LuMp, 23sp) with the curation of two new large-scale benchmarks (LuMp, 23sp). Below, we address the specific questions raised in reviewer 23sp's comments.

(23sp-Q1) What happens when SLAM struggles?

Since our approach utilizes semidense keypoints derived from SLAM, their quality could affect performance. However, this applies to any framework that relies on SLAM-derived keypoints as input. Still, it is feasible to utilize accurate pre-mapped information as a fallback option since our framework should typically be deployed in familiar everyday environments, e.g., home and office.

To conduct a sensitivity analysis examining how the performance of our framework changes when SLAM struggles, we explore several types of sensory corruption across mild to severe scenarios. First, multi-sensor devices may experience temporary frame drops or time drift due to hardware issues, representing temporal corruption. Second, spatial degradation in egocentric localization may occur, which we apply separately to translation and rotation components. Finally, we consider corruption that can affect the set of keypoints by adding Gaussian noise to individual points or dropping certain points entirely. Performance results measured on the FoVS-Aria test split for each corruption type are presented in the following table:

Under mild corruption, the impact on performance is negligible. However, applying higher-intensity corruption results in performance degradation of several percentage points. Among the different spans, those requiring wider coverage (Orientation and Peripheral) show smaller performance drops. In contrast, spans demanding higher precision (Central and Foveal) exhibit larger degradation. Despite these challenges, our framework maintains superior performance compared to EgoChoir even under severe corruption scenarios. This suggests that our framework can perform reliable forecasting despite some degree of SLAM-induced imprecision. In practice, real-world scenarios involve multiple types of corruption occurring in complex and somewhat unpredictable combinations. Therefore, training our framework to be robust against such corruption would represent an interesting extension for future work.

(23sp-Q2) Detailed analysis on the latency of our framework

Our framework has two primary sources of latency: (i) extracting the relevant set of points from gaze and SLAM keypoints (performed every 100ms), and (ii) performing model inference every second on a set of points spanning two seconds.

Since the point extraction can be pre-computed and stored at 10 fps for continuous use, we only need to consider the computation time for processing the final observation when calculating inference latency. We measured this in a resource-constrained environment compared to our training setup, using 8 CPU cores and a GPU with 12GB VRAM (e.g., Titan X). In the table below, all operations except model inference are processed on the CPU, with the following execution time:

The first stage, which handles outlier removal, axis-aligned bounding box cropping for keypoints, and selection of points within a certain degree of eccentricity from the gaze, can be processed within 10ms. In the second stage, we identified that the primary bottleneck lies in voxelization rather than in the model itself. This occurs because the large number of keypoints from the previous stage should be voxelized, whereas the model operates efficiently once it receives the 3D voxelized representation.

Consequently, the average inference latency is 71.241ms, yielding a real-time factor of 0.036 when processing 2-second input for multi-second forecasting. This confirms our claim that the framework supports fast processing. However, actual AR/VR environments typically operate with even fewer computational resources, and thus additional optimization techniques such as model quantization or more efficient voxelization could be considered for further performance improvements.

23sp, please could you take a look at the author response above and whether it addresses any remaining concerns you have, e.g. the requirement of a good 3D SLAM map.

We thank reviewer k89X for the detailed comments. To summarize all reviewer feedback, we are encouraged that the reviewers find our problem formulation interesting and advantageous (7cjL, LuMp, 23sp) while being methodologically solid and innovative (7cjL, LuMp) with clear writing (7cjL, k89X). Additionally, they find that our comprehensive experimental results demonstrate significant performance improvements (7cjL, LuMp, 23sp) with the curation of two new large-scale benchmarks (LuMp, 23sp). Below, we address the specific questions raised in reviewer k89X’s comments.

(k89X-Q1) Conceptual motivation of our work

As noted in L39-40, we acknowledge that the term 'visual span' originates largely from text and symbolic reading research in cognitive psychology. While we draw inspiration from this foundation, our work addresses daily and casual behaviors and interactions from an egocentric perspective, without aiming to provide precise computational modeling of the original terminology. While reading-based visual span depends on specific textual stimuli, forecasting visual perception in egocentric daily activities involve diverse, dynamic environments where stimulus-independent spatial regions provide a more practical and generalizable framework. We will clarify the terminology discrepancies in our final draft. Please note that the other three reviewers have recognized the merit of this conceptual formulation, with reviewer 7cjL highlighting our 'practical problem formulation', reviewer LuMp noting that we are 'clearly motivated by cognitive theories and real-world applications', and reviewer 23sp stating that we 'frame the problem in a sensible fashion that provides many advantages,' respectively.

Our cone-based multi-level interpretation serves as a plausible and practical representation for modeling eccentricity and periphery, which is reflected in our substantial performance gains over prior art. We note that gaze, visual focus, and visual span are known to be interconnected [VisCog’13, TPAMI’18, NatHumBehav’19], and similar concepts and wordings are commonly used in related computer vision work, including previous research [16, 17, 32, 33, 34, ETRA’10, ICCV’21]. Importantly, our core methodological and empirical contributions, which have been appreciated by the other three reviewers, remain valuable regardless of naming choices. To address any remaining terminological concerns, we are open to adopting alternative terms.

(k89X-Q2) Rationale behind our multi-level representation

While the monotonic results in Table 1 across different spans may appear redundant, this impression arises simply from the broad prediction target regions. In reality, multi-level interpretation carries significant implications due to uncertainties in self-motion forecasting and geometric correspondence between 2D and 3D spaces. To demonstrate this importance, we conducted an ablation study applying post-processing to 3D gaze forecasting using a model trained with the multi-level components removed from our framework:

Predicting only gaze and applying post-processing is clearly suboptimal. Compared to the foveal forecast performance when jointly predicting all multi-level spans, foveal forecast performance shows an F1 decrease of approximately 5.2, stressing the importance of our multi-level representation and multi-task objectives. Furthermore, the central forecast exhibits an F1 decrease of approximately 9.7, which is substantially larger than the 2.8 F1 performance decrease observed in our model trained to predict only the central span. This demonstrates that predictive performance across different spans involves complex geometric and learning dynamics that extend beyond simple monotonic relationships. We will include this analysis in our final draft to clarify the benefit of our formulation.

(k89X-Q3) Suggestions on relevant papers

Thank you for your suggestion regarding the related work from mobile and VR research. We will cite these works [MobileHCI'18, TVCG'21, GCPR'22] and revise our discussion accordingly. Please note that we reference works [24, 25] in our related work section before narrowing our scope to egocentricity, as these are some of the seminal works for gaze comprehension in computer vision to the best of our knowledge.

Reference

[ETRA’10] Blignaut (2010). Visual Span and Other Parameters for the Generation of Heatmaps.

[VisCog’13] Nuthmann (2013). On the Visual Span during Object Search in Real-world Scenes.

[TPAMI’18] Masse et al. (2018). Tracking Gaze and Visual Focus of Attention of People Involved in Social Interaction.

[NatHumBehav’19] van Ede et al. (2019). Human Gaze Tracks Attentional Focusing in Memorized Visual Space.

[ICCV’21] Jiang and Ithapu (2021). Egocentric Pose Estimation from Human Vision Span.