PARSE-Ego4D: Personal Action Recommendation Suggestions for Ego-Centric Videos

We thank the reviewer for their thoughtful feedback and helpful suggestions, which we believe will significantly enhance the quality of our work. We also appreciate the reviewer’s recognition of the potential impact of the implicit query-to-action task, as it leaves substantial room for future advancements in the field.

• W1 - single annotation for 80% of dataset: due to budget constraints and our commitment to paying all annotators above the U.S. minimum wage, we prioritized collecting extensive annotations (5 per sample) for 20% of the dataset while using single annotations for the remaining 80% for training. This tradeoff balances resource limitations with the practical need for larger datasets. Real-world data often contains noise, and foundation models’ inherent world knowledge allows them to generalize well even with noisier annotations. Expanding the dataset with higher-quality annotations remains an important avenue for future work.

• W2 - action subdivisions: we acknowledge the imbalance in action types and agree that subdividing broader categories like "instructions" and "assistants" would improve the dataset's utility. This effort is currently underway, and we aim to provide updates in future versions of this work.

• W3 - classification task: the 87% accuracy was achieved using a state-of-the-art, fine-tuned, open-source small language model with 2B parameters (Gemma2-2B). This represents a competitive result, significantly outperforming random and zero-shot baselines. While promising, it still leaves ample space for improvement, particularly with larger and more diverse datasets or novel architectures.

• W4 - dataset size and quality: we agree that 7770 high-quality samples is relatively small by modern standards. However, given our budget constraints, we chose to release PARSE-Ego4D as a starting point with an open methodology for reproducibility and scaling. The code for generating and filtering the dataset, as well as for the ML baselines, will be open-sourced (see response to Reviewer xC2F). As progress in multimodal foundation models accelerates (see shortcomings of multimodal FMs in our response to Reviewer xC2F), we anticipate scaled-up versions of this dataset in the near future to complement these advancements and provide even greater utility. We also note that it is possible to lower the threshold, depending on the task, in order to have more samples to train on.

• W5 - NLL metric for Task 2: we agree that the metric of evaluation for Task 2 is important. However, the metrics proposed by the reviewer are not suitable for our task. The main problem for adopting these metrics is that our proposed Task 2 does not have a single, correct reference answer. There are multiple viable recommendations. Our dataset contains one, or even multiple, examples for an action suggestion for a given context, but we cannot claim that this is the only correct answer. WUP is a similarity metric that compares single words based on some taxonomy - we are dealing with sentences and there is no clear choice of a taxonomy to be used in our context. BLEU, ROUGE and METEOR evaluate the alignment of some text with a single correct reference text. The additional stemming and synonymy features of METEOR are relevant to us, but they do not solve the problem of having multiple correct, yet semantically distinct, answers. For this reason, we chose to only use the negative log-likelihood (NLL) to evaluate the performance on Task 2. The NLL evaluates the likelihood of the model generating a suggestion from our dataset without penalizing the model for potentially also generating other valid suggestions.

• W6 - Continual Interactions: the reviewer raises an excellent point regarding continual interactions. We briefly touched on this point in our discussion section (Multi-turn suggestions and bespoke UI, line 515 in the revision), but we have expanded this further to highlighting it as an important direction for future work.

• Question: yes, all ML experiment results are based on the dataset filtered with the {sensible, correct} ≥ 4 threshold, as specified in the paper.

We hope these clarifications address the reviewer’s concerns and provide additional context on our methodology and its tradeoffs. Thank you again for the constructive feedback and for identifying opportunities to refine and expand this work. We respectfully encourage reconsideration of the rating in light of these responses.

We thank the reviewer for their thoughtful feedback and suggestions. Below are our responses to the key points raised:

• Dataset availability: the dataset is already publicly released, ensuring accessibility for researchers. There is no uncertainty regarding its availability.

• Template issue: while we acknowledge the template oversight (my sincere apologies), we have corrected it and assure the reviewer that all prior peer review feedback, including from NeurIPS, has been carefully addressed.

• Focus on benchmark tasks: we included a related work section for the benchmark tasks and have expanded the description of their novelty and significance in the revised manuscript, see Line 144++.

• Implicit task: we admit that the section was not very concisely written. We rewrote all of Section 4 to clarify the benchmark, add more details on the models, and include explicit references to figures to better illustrate the proposed tasks. We hope that this revision greatly improves the presentation and clarity of our work.

• Q1: indeed, the "others" category was omitted from Task 1. This omission has been clarified in the revised manuscript (last sentence of the first paragraph in section 4.1).

• Q2 – NLL metric: this point is already addressed in the original manuscript (lines 291–295). The goal of Task 2 is to generate contextually appropriate action suggestions. We recognize that evaluating generated suggestions can be challenging due to the potential for multiple valid actions in any given scenario. Metrics like BLEU scores assume a single “correct” reference output and hence do not capture the nuances of different viable actions. So we used the negative log-likelihood (NLL) of generating a (query, action) pair that human annotators have rated highly. This approach aligns more closely with real-world scenarios, where the diversity and quality of suggestions are crucial.

• Q3 – baseline model choices: our reported baselines were chosen for their simplicity and relevance: zero-shot baselines used public APIs of popular foundation models, and trained baselines utilized open-source models fine-tuned with standard ML practices. The table in the revised manuscript is now clearer, with missing entries explained in the caption (due to either (a) unavailable information for closed-source models or (b) unsuitability of certain models, such as embedding + MLP models, for generating text that is necessary for Task 2).

• Q4 – single annotations: due to budget constraints and our commitment to paying all annotators above the U.S. minimum wage, we prioritized the reliability of our evaluation and test datasets, recognizing that some noise in the training data is inevitable. Our approach appears effective, as evidenced by consistently better model performance on the test dataset compared to the training dataset.

We hope these clarifications address the reviewer’s concerns. The paper builds on an impactful dataset and proposes meaningful tasks that we believe will foster further research in assistive AR. We kindly encourage the reviewer to reconsider their score in light of these updates. We believe that we have been able to improve the presentation of our contributions significantly, thanks to the helpful feedback from all reviewers. Thank you again for the constructive feedback.

We appreciate the thoughtful feedback and suggestions from the reviewer. Below are responses to the points raised:

• Qualitative results (updated): as the reviewer acknowledges, we focused on quantitative analyses. It is not clear to us what qualitative results the reviewer expects us to provide. Figure 6 in the Appendix shows the histograms of action categories that are accepted or rejected, and includes a short discussion of our findings from this data. If including more representative samples for accepted and rejected suggestions would be helpful, we can easily provide these in the Appendix of the camera-ready version. We could also analyze the "failure modes" of the generating LLM, by giving our interpretation for why certain suggestions were rejected (note that we did not ask annotation participants for a free-form explanation of their ratings, so we can only hypothesize on the reasons for rejection/acceptance).

• Surveying actors: collecting a new dataset with actor-specific survey responses is beyond the scope of this work. Instead, this study builds on the widely accepted Ego4D dataset, which provides community-endorsed annotations and is already extensively used. Recreating such a dataset would be prohibitively expensive and impractical for us, and leveraging Ego4D ensures our work is accessible and builds upon a robust foundation.

• Multimodal baselines with video input: incorporating video input into baselines presents significant challenges. Our preliminary experiments required selecting representative frames, as feeding continuous video (e.g., 30 fps) is not feasible for hour-long streams of egocentric video. This is compounded by computational constraints, particularly for AR use cases which require small and efficient models. To balance efficiency, engineering complexity, and maintainability, we used textual narrations—a well-established proxy for video content, using the verified annotations from the original Ego4D dataset. This tradeoff enables practical model deployment while leaving room for future multimodal research. We hope the reviewer recognizes that this tradeoff aligns with practical constraints and enhances the utility of our contribution.

• Code Availability: the code for generating the dataset will be released as part of the GitHub repository accompanying the camera-ready version, ensuring the reproducibility and extensibility of this work. We have already open-source released the ML benchmark evaluation code, and are cleaning up the data generation code for external use.

• Clarification on Cell Phone Search Data: the dataset does not include data on humans searching on cell phones, and such descriptors are neither part of this work nor discussed in the paper. The reviewer’s suggestion appears unrelated to this study. We would encourage the reviewer to clarify their question so we can better support this suggestion.

• LLM Recommendations for Cognitive Actions: we are still analyzing whether LLM-generated recommendations perform better for cognitive actions and will address this in future updates. update: please see the follow-up below for a proposed study of this suggestion.

We hope these clarifications highlight the soundness of our methodology and the practicality of our approach within the constraints of the field. We believe the paper makes a meaningful contribution to egocentric AI research and respectfully encourage the reviewer to consider revising their score in light of these clarifications. We have significantly improved the presentation of our contribution by incorporating feedback from all reviewers.

We again thank the reviewer for their time and suggestions on improving our paper.

We thank the reviewer for their feedback. Here are some clarifications:

Human-derived narrations: the narrations grounding our action suggestions come from human viewers describing the videos, providing realistic and contextually relevant perspectives. While not flawless, these human observations offer a meaningful base for developing proactive action recommendations. Future work could explore additional grounding, but we propose this dataset as a valuable initial step.

Temporal grounding: as noted in the manuscript (line 182), each narration covers ~2 seconds of video, sufficient for generating relevant action suggestions. Small timing shifts (±5 sec) are unlikely to impact the usefulness or relevance of these suggestions in a real-world setting, especially for egocentric applications.

We hope this clarifies our approach and highlights the significant value and potential of our dataset. We kindly encourage the reviewer to consider this additional context when evaluating the contribution and impact of our work.