EgoPrivacy: What Your First-Person Camera Says About You?

>>> Q1 Key factors that affect the performance of RAA.
>>> A1 Thanks. In fact, we do provide an analysis of these factors in Appendix B and Figure 6, due to limited space. We show that, overall, soft voting with 𝑤=0.5 yields the best performance, and varying the top‑k retrieved exo-predictions in RAA shows a performance plateau around k=3 or k=5 due to inaccurate additional exo-views, and including the ego prediction improves the attack.

>>> Q2 Comparisons with different input frames are not fair.
>>> A2 Thanks. We agree that comparing with the frame number is not entirely fair. Our original intention was not to make direct comparisons, but rather to showcase the best attack performance of each model within the same computational constraints (8xRTX 4090 for training). We here provide a more equitable comparison by using the same number of frames across all models.

Although EgoVLPv2 performs worse than CLIP with 8 frames, it offers better computational efficiency due to its early fusion, allowing for improved performance with more frames, thus achieving the best attack results while balancing performance and memory consumption. We will also discuss this point in response to Q7. We will add theresults to Table 3.

Regarding temporal modeling, we acknowledge it is not sufficiently rigorous. As noted in Section 6.2, we position this as only a preliminary observation due to the limited time and space. We present a more thorough investigation by applying RNN, Attention (with pos emb) and MLP of similar param size on top of CLIP (see https://imgur.com/a/YMXXZMx). We show that both gender and race attack benefits from temporal modeling, while age is the exception.

>>> Q3 Only 4 to 5 vision-language models are included for comparison, which is insufficient.
>>> A3 Thanks for the comments. While we appreciate your concern, we believe that 4 to 5 models are sufficient for a comprehensive study in this context. To address your concern within the time constraint, we additionally add Qwen2.5-VL [1] results below:

>>> Q4 Missing related work [1,2,3,4]

>>> A4 While cross-view retrieval work [2, 4] exists, its application to egocentric demographic privacy is limited—[1] emphasizes semantic retrieval for action captioning, and EgoTransfer [3] and ObjectRelator focus on motion and object recognition, whereas our method targets instance-level retrieval for privacy attacks; we will include a detailed comparison in the final version.

>>> Q5 typos
>>> A5 Thanks! We will revise all typos in the final version of the paper.

>>> Q6 How is zero-shot eval employed in VLM-based models (LLaVA and VideoLLaMA2)?
>>> A6 We leverage the ability of VLM to output free-form texts and format the privacy attack as open-ended VQA.
An example of the prompt:

---

This is a video taken by a wearable camera. What is the gender of the wearer? A. Female B. Male Answer with the letter of the correct option.

---

We use template matching to extract the answer, as all VLMs reliably follow instructions, making it sufficient for result extraction.

>>> Q7 EgoVLPv2 is trained using 4/16 frames, while the authors adopted 128 frames for evaluation.
>>> A7 Indeed, EgoVLPv2 we adopt only accepts 4 frames as input, but videos in the EgoPrivacy benchmark contain 1000+ frames. To address this, we split the 128-frame input into 32 chunks of 4 frames, processed independently by EgoVLPv2. The results are then aggregated via a classification/retrieval head. For zero-shot evaluation, we score and ensemble all 32 chunks' outputs to generate the prediction. We found 128 frames to offer the best balance of performance and efficiency. Following the suggestion, we perform the experiment as mentioned above with only 8 frames to provide a fair comparison. This resulted in a slight decrease in performance, which supports our choice of 128 frames for evaluation.

First of all, we sincerely thank the reviewer for the valuable feedback and for acknowledging that our work haspromoted privacy protection in the first-person perspective. We will address the concern below.

>>> Q1 There is a missing comma in Equation 4. At the end of line 253, "but as aa video" should be "but as a video". "Appendix ??" in line 438 needs to be corrected. The parentheses in image (d) of Figure 1 need to be fixed.
>>> A1 Thanks for your comment. We will revise the paper and fix all typos in the final version.

>>> Q2 The author defines three types of privacy: Demographic privacy, Individual privacy, and Situational privacy. Regarding whether there are other privacy issues in the EgoPrivacy benchmark, for example, the videos shot by users may contain some privacy-leaking information such as bank card numbers.
>>> A2 Thanks for the insightful comments. This is absolutely a significant privacy concern. However, our paper focuses specifically on demographic, situational, and individual privacy, which are directly related to the person in the context of egocentric videos. Thus, the mentioned privacy issue, such as bank card numbers in egocentric videos, falls out of the scope of this paper. However, we cite relevant papers [1] for the reviewer's reference and consider this as a potential and important direction for future research.

[1] Raina, Nikhil, et al. "Egoblur: Responsible innovation in aria." arXiv preprint arXiv:2308.13093 (2023).

>>> Q3 Figure 1 is difficult to understand. Most of the images are from the egocentric perspective, and there are a few from the exocentric views. It's unclear what information the images are trying to convey. It is recommended that the author polish Figure 1.
>>> A3 Thanks for the comment. Figure 1 is indeed a bit misleading. The goal of this figure was to illustrate different types of privacy risks: (a)-(c) represent demographic privacy, (d)-(e) represent identity privacy, and (f)-(g) represent situational privacy, as defined in Section 3.1. All input examples are presented in the egocentric view since the primary focus of this work is to investigate egocentric privacy. For retrieval-based privacy tasks (identity and situational), we consider both ego-to-ego retrieval (where both the query and target are in the first-person view) and ego-to-exo retrieval (where the target is in the third-person view).

To improve clarity, we will annotate the two perspectives in the figure and include a more detailed explanation in the introduction.

Again, thank you for the valuable suggestions, and we hope that our rebuttal has addressed all the concerns proposed. Please do not hesitate if you have any further questions or concerns. We look forward to your response and valuable opinions!

>>> Q1 I would also like to see a short discussion on potential mitigation strategies to counteract this type of privacy attack, based on the insights gained from these experiments.

>>> A1 This is a great suggestion. While we intend to leave the study of privacy risk mitigation for future research, we will include additional discussion on the potential direction to solving this problem. First, we expect methods designed to protect third-person privacy (e.g., face blurring [1]) to be less effectve, as it does not address the privacy of camera wearer. Second, segmenting and obfuscating body parts (e.g., hand and feet) can reduce privacy risk, but also makes utility tasks such as egocentric action recognition much harder. As such, an adaptive method for processing the egocentric vidoes is needed for the optimal trade-off between privacy and utility.

In addition to the post-hoc techniques for software-level privacy preservation, another promising direction is to design cameras that can achieve hardware-level privacy-preservation, which further eliminate the privacy risk caused by the leak of raw camera measurements. The concept of privacy-preserving cameras has attracted attentions in the community [2,3,4], but their application in the egocentric setting remains to be investigated.

[1]EgoBlur: Responsible Innovation in Aria. Arxiv 2308.13093. [2]Learning privacy-preserving optics for human pose estimation. ICCV'21. [3]Learning a Dynamic Privacy-Preserving Camera Robust to Inversion Attacks. ECCV'24.

>>> Q2 Could the authors clarify which models in Tables 2 and 3 correspond to these four threat models? A clearer way to distinguish and denote different types of models would improve readability.

>>> A2 Thanks for the question and the advice. We will first clarify the correspondence between threat models and the results in Table 2 and 3. Then, following your suggestion, we will introduce a new table that compares the performance across different threat models. This new table will replace part of Table 2 to enhance clarity, and we will split Table 2 into smaller, more focused tables in the camera ready version.

To clarify:

• Zero-shot results correspond to Capability 1. These results are shown in the first sections of Table 2 (excluding the RAA results). In Table 3, the rows marked with "ZS" correspond to the Capability 1.

• Fine-tuned models belong to both Capability 1 and Capability 2. Keep in mind that Capability 1 is the basic capability, as all attacks require some form of query and model. In Table 2, the "Fine-tuned" section (second section) represents results related to Capability 1 and 2 (excluding RAA results). In Table 3, these results are marked with "FT."

• RAA results:

  • In the Zero-shot section (first section) of Table 2, RAA corresponds to a composite of Capabilities 1 and 3, as it combines a base non-finetuned model with the ego-exo retriever and a pool of exo-videos.

  • In the Fine-tuned section (second section) of Table 2, RAA is a composite of Capabilities 1, 2, and 3, since it involves fine-tuning in addition to the ego-exo retriever and exo-video pool.

We hope this clarifies the relationship between threat models and their corresponding results in the tables.

Here, we present the new table directly compares all the threat models. Due to space limit, only the results for the CLIP model is displayed here.

>>> Q3 In Table 2 that for Gender and Race, "Exo" consistently outperforms "Ego" before and after fine-tuning. However, for Age, after fine-tuning, "Exo" performs worse than "Ego". Could the authors provide insights into why this occurs?

>>> A3 We believe this occurs due to the inherent subjectivity and complexity of age recognition. Unlike gender and race, which often have more distinct visual cues, age estimation is influenced by various factors such as lighting, facial expressions, and individual aging patterns. The fine-tuning process may have led the model to overfit to certain biases present in the egocentric data, resulting in better performance for ‘Ego’ compared to ‘Exo’ after fine-tuning.

>>> Q1 Theoretical guarantees or formal proofs of retrieval Bound

>>> A1 Thanks for the insightful comment. This paper indeed focuses more on empirical evidence of egocentric privacy risks, where measured by hit rate among top $K$ retrievals. This means we do not predefine thresholds for matching videos of the same identity, but use the k-th largest score among the candidate examples as an adaptive threshold. This implies a larger threshold as the number of idendidites in the test set increase, and hence a lower hit rate (equivalently, higher probability of false matches).

Concretely, assuming a retrieval setting with $N$ individuals and $M$ candidate examples per identity, the chance-level hit rate @ $K$ is given by $\textrm{HR}_\textrm{chance}@K = 1 - \frac{\binom{M(N-1)}{K}}{\binom{MN-1}{K}}.$ This approaches $1 - (1 - \frac{1}{N})^K$ when $M$ is large and $\frac{K}{N}$ when $N \gg K$, as is usually the case for large-scale datasets.

>>> Q2 How do false matches (collisions) impact method performance after aggregation? False-Match Considerations: Do you have any observations on how often RAA might retrieve irrelevant exocentric videos when the embedding space has many visually similar but semantically mismatched samples?

>>> A2 Great question! Yes, it is entirely possible that RAA may be misled by false positives in retrieval. The retriever used in RAA is based on the ego-exo retrieval model from EgoVLP v2. As shown in Table 3, this model achieves a 50.31% HR@1 and 66.82% HR@5. However, we empirically find that this relatively low retrieval rate does not significantly impact the demographic prediction accuracy, as discussed in the second paragraph of the "Effectiveness of RAA" section (Section 6.2).

Despite the lower retrieval accuracy, RAA still enhances the attack's effectiveness. We hypothesize in L376-427 that even with imperfect retrieval, the process helps cluster and group identities with similar demographics (such as gender, age, and race) closer in its learned embedding space. This is evidenced in Table 4, where, despite a 50% retrieval accuracy, 82.22%, 84.51%, and 82.95% of the top-1 retrieved exo-videos share the same gender, age, and race as the ego-video, respectively. This suggests that even negative retrievals can benefit RAA if they are semantically close to the query.

>>> Q3 Minor typographical error at line 437 in the main text.

>>> A3 Thank you for pointing this out. We will revise the paper and fix this typographical error.

>>> Q4 What is the aggregation mechanism?

>>> A4 Thanks for this good question. In fact, we explore various aggregation mechanisms in Appendix B and Figure 6. The default mechanism involves a weighted sum of both the ego and exo predictions (referred to as soft voting), where the ego weight is set to 0.5, and the remaining 0.5 is evenly distributed across the exo predictions. In addition to this, we also examine hard voting (direct majority voting) and soft voting with equal weights assigned to all predictions. Our experiments show that, overall, soft voting with 𝑤=0.5 yields the best performance, improving all three demographic tasks consistently compared to the ego-only baseline (denoted by the dashed line).

Additionally, we investigate the impact of varying the number of top-k retrieved exo-predictions used in RAA. Our findings reveal that there is a key pivot point around 𝑘=3 or 𝑘=5, beyond which increasing k does not improve performance. This is because additional retrieved exo-views often introduce noise due to their potential inaccuracy. We also perform an ablation study on the role of the ego prediction in RAA, showing that including the ego prediction generally enhances the attack's effectiveness. We also experiment with using the retrieved similarity as a confidence measure to weigh each prediction, however, this empirically fial to improv over even the majority voting.

>>> Q1 Although the authors effectively demonstrate demographic leakage using general-purpose foundation models (e.g., CLIP, LLaVA), the reported performance likely underestimates the actual privacy risks.
>>> A1 Thanks for the suggestion. While models tailored to privacy attacks may have an advantage, there's a significant gap between datasets used for facial attribute prediction—typically high-quality, cropped facial images (e.g., CelebA, UTKFace, FairFace)—and EgoPrivacy, which uses raw, distant egocentric video (see image left). We argue that SOTA models perform well in ideal conditions, but their metrics aren't directly comparable to results on EgoPrivacy.

We also agree that current vision-language foundation models, despite performing surprisingly well, aren’t optimized for egocentric privacy tasks. Their performance could improve with more egocentric training data. This motivates our benchmark: to drive research, enable objective progress tracking, and advance attack and defense methods in the egocentric privacy domain.

>>> Q2 For egocentric classification, the authors do not explore hand-based biometric models.
>>> A2 Just as exocentric video frames with faces do not resemble images from high-quality facial attribute datasets (see A1 above), the distribution gap between hand/palmprint identification data and egocentric videos is even more dramatic, as shown in image (right). The vast majority of egocentric video frames do not show clear pictures of the hands of the camera wearer; even those that do tend to show the back of hands from first-person view, which contains far less identifiable information than the palm. This makes it very challenging to apply hand biometric methods to egocentric privacy tasks directly without nontrivial modifications or retraining. We will include a thorough discussion of specialized approaches to egocentric privacy attacks in the final version of the paper, as well as empirical comparisons to hand-based baselines.

Regarding additional baselines in exocentric (face-based) and egocentric (hand-based), we have invested effort to reproduce the results of the suggested papers on our benchmark. However, due to the limited time and computation, as well as the unavailability of pretrained models in public and the scale of our evaluation set (over 5000 exo- and ego videos), we are unable to offer results at the moment, but hopeful can share results during the discussion phase and final version of the paper.

>>> Q3 Limited novelty in terms of data collection.
>>> A3 While we do repurpose Ego-Exo4D and Charades-Ego dataset for egocentric privacy, additional effort has been made on the annotation of demographic labels, on an unprecedented scale, which is a nontrivial task. This annotation further facilitates the significant research on the egocentric privacy that has received limited attention in prior egocentric vision literature.

>>> Q4 The proposed Retrieval-Augmented Attack (RAA) closely resembles earlier cross-view retrieval methods, such as those by Fan et al. (2017) and Ardeshir et al. (2018), and does not introduce significant algorithmic innovation.
>>> A4 Thank you for your comment. As discussed in Section 2, "Related Work on Egocentric Person Identification," we acknowledge the contributions of earlier cross-view retrieval methods [1, 2, 3, 4]. However, previous work has largely overlooked the impact they have on the demographic privacy of camera users, i.e. how cross-view retrieval techniques might facilitate security breaches of egocentric demographic privacy, which is the primary focus of our study.

For instance, [1, 2] focus on identifying first-person camera wearers in third-person videos, but they do not explore the potential of these methods in enhancing demographic attacks. Similarly, [3] investigates cross-view techniques for retrieving motion features, while [4] uses cross-view retrieval to improve action and motion captioning at the semantic level, neither of which are designed with privacy concerns in mind. In the final version of the paper, we will include a more detailed discussion of the novelty of RAA in relation to these prior works.

[1] Fan, Chenyou, et al. "Identifying first-person camera wearers in third-person videos." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017. [2] Elfeki, Mohamed, et al. "From third person to first person: Dataset and baselines for synthesis and retrieval." arXiv preprint arXiv:1812.00104 (2018). [3] Ardeshir, Shervin, Krishna Regmi, and Ali Borji. "Egotransfer: Transferring motion across egocentric and exocentric domains using deep neural networks." arXiv preprint arXiv:1612.05836 (2016). [4] Xu, Jilan, et al. "Retrieval-augmented egocentric video captioning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

>>> Q1 Calculation of random accuracy

>>> A1 Thank you for the insightful question. We present the prior accuracy

However, imbalance also poses challenges during training. Thus, we also perform additional studies that merge / remove the extremely imbalance category in the age / race demongraphic attacks. Specifically, we merge the old with the middle age in age and remove the black in the race, as these two categories are extremely minority in the dataset. We present additional results below:

In general, we observe that gender and race attacks outperform prior accuracy by large margins, while the improvement of age attacks is less significant due to heavy imbalance in age demographics. We leave for future work the collection of more balanced egocentric video datasets for an unbiased evaluation of demographic privacy.