ALBAR: Adversarial Learning approach to mitigate Biases in Action Recognition

W4: Although the paper proposes a simplified end-to-end training framework, adversarial training often involves additional computational costs. The paper does not discuss the computational efficiency and scalability of the ALBAR method in detail.

• Response to Weakness & Question 4: This is a good point for us to address. While adversarial training does introduce computational overhead, our framework is intentionally designed to minimize additional computational costs. Specifically, the only additional cost comes from creating the static video batch and passing it through the model in addition to the standard clips, which scales only linearly with the video encoder size. Critically, we avoid loading additional models, introducing new parameters, or overly complex adversarial architectures. On HMDB51, using a single 80GB A100 GPU, our complete training process (including validation each epoch) requires approximately 9 hours—demonstrating the method's computational efficiency. With the only additional overhead coming from the computation of extra losses coming from the same model, our approach is lightweight and practical. Apart from the training computational cost, it is important to note that our method incurs no additional computational overhead during practical deployment compared to a standard model.

Thank you for your insightful review and detailed questions.

W1: Although the results demonstrate that the model outperforms state-of-the-art methods, the technical contribution is incremental and purely based on previously introduced techniques.

• Response to Weakness & Question 1: While the basic idea of adversarial training, entropy maximization, and gradient penalties are not alone novel, we want to highlight the aspects of our technical contribution that are novel:
  1. First, adversarial training typically utilizes labels or separate models to facilitate the counter-objective, while we reduce complexity and create a strong negative by utilizing the same encoder and classifier head, manipulating the input itself instead. By consolidating adversarial training techniques into a single, streamlined framework, we offer a more efficient and integrated approach to addressing bias in video action recognition.
     • Previous works have utilized separate 2D and 3D encoders, repelling representations in a contrastive-like objective. We are the first to combine the objective into the same encoder and adversarially train in this manner.
  2. Our formulation of minimizing gradient norm to stabilize adversarial training weight updates w.r.t. a specific input type represents a novel technical contribution. This approach provides a more nuanced method of managing adversarial training dynamics.

W2: How is the fairness of the comparative experiments ensured in this paper, and how are the results of the comparative methods obtained?

• Response to Weakness & Question 2: Great care was taken to ensure fairness in experimentation. Most results were sourced from prior publications. The StillMix paper released their codebase along with comprehensive hyperparameter choices. Due to this, we were able to replicate their numbers in our own implementation. We retained their original hyperparameter choices and incorporated our proposed losses, ensuring a fair comparative evaluation. In the case of results not being reported previously—on our proposed UCF101 protocol fix—we used the reproduced model for evaluation.

W3: The ALBAR performs well on specific bias evaluation protocols, but its generalization capabilities to new types of biases or different domain tasks have not been fully validated.

• Response to Weakness & Question 3: We appreciate your point here. In the following Table 1, we demonstrate our method's generalizability across different domain tasks (weakly supervised anomaly detection, temporal action localization). Regarding biases, the SCUBA/SCUFO protocols are designed to comprehensively address static biases in video action recognition, notably introducing foreground bias evaluation. Currently, we are unaware of alternative protocols for assessing other types of video action recognition bias. We welcome suggestions for additional bias evaluation methods and would be eager to incorporate them. In the table, HMDB51 (OOD) refers to the Contrasted Accuracy (Contra. Acc.) results explained in Main Paper Section 4.4. The downstream tasks use features extracted from our debiased model trained on HMDB51. Notably, performance is greatly improved across tasks that require high-quality temporal understanding. These results highlight that our approach is not only relevant for action recognition but also beneficial for diverse downstream tasks in video understanding, further establishing its broader impact. For additional details on this implementation/analysis, please refer to our initial response to Reviewer EAVn.

Table 1: Additional video understanding task results comparison between baseline model and one trained with our ALBAR framework. Action recognition performance is reported as Top-1 accuracy (%), anomaly detection score is given as AUC (%), and temporal action detection as mAP (%). A higher score is desired for both tasks.

W2: Does the "static clip" way of approaching video bias mitigation, have broader utility in video understanding or beyond?

• Response to Weakness 2: Video biases, such as foreground or background bias, primarily stem from the static appearance cues in a video. These biases arise due to the spurious correlation between the appearance and the action label, leading to predictions that rely on appearance rather than motion. By using a static clip, which inherently eliminates motion information, we ensure that the video encoder generates features solely based on the static appearance of the frames. Penalizing the model for making predictions based on these static appearance features helps mitigate appearance-based biases, including foreground and background biases. Since static clips exclusively capture appearance-related information, this approach can be broadly useful in mitigating all types of biases that originate from appearance, extending its utility beyond just action recognition to other video understanding tasks.

W3: Certain interesting experiments are omitted, even though the writing suggests they may be useful. For example, L233 reads "A naive application of Eq. 2 results in degraded performance." What exactly is a naive application? Furthermore, this implies that the two parts of Eq. 2 is individually interesting; in particular, does the right hand side do anything? The ablation study does not include these two parts separately. Q5: How are there reasonable IID results in Table 3 for the case that Ladv is not used ---> when it is not used, there is no actual gradient to guide the model to do any recognition?

• Response to Weakness 3 & Question 5: This confusion appears comes from our non-ideal notational choices for Ladv In all experiments, we include the cross entropy loss on the standard motion clip (the left hand side of Ladv as written). In cases where Ladv is not applied, technically this just means that omega_adv = 0 (the right hand side weight). We grouped these components into a single equation to illustrate their adversarial relationship, acknowledging this approach complicates future notation, though we acknowledge that this makes future notation tricky. To improve clarity, we propose separating Ladv into distinct losses and will adjust notation accordingly throughout our work. This modification will help prevent misinterpretation and make the mathematical formulation more transparent.
  • Weakness 3 part 1: Naive application may not be the best wording here. This simply means applying the loss (Ladv) without the additional regularizing losses (Lent, Lgp).
  • Weakness 3 part 2: Technically, the ablation that evaluates the left hand side separately from the right hand side of Ladv is Main Paper Table 3 row (a) vs. row (b). We did not ablate using just the static adversarial component without the base temporal cross-entropy, though doing so just results in the model devolving into a state where it predicts the same class for every input, technically achieving 1.96% on IID HMDB51. Addressing notation should clarify this point.
  • Question 5: To directly address Q5, the baseline L_CE is still utilized in situations where Ladv is not, just not the right hand static adversarial side (omega_adv = 0).

Q2: The method works by sampling any frame from the video and repeating it for a static clip. Why any frame? Aren't certain frames better or worse than others for the stated goal? (The pre-segmented nature of the datasets in question create itself an algorithmic bias. In the stated real world application deployments no such pretemporal segmentation is avavilable, and hence brings into question the feasability of the method in practice.) But, more concretely to the task, why is there no analysis whatsoever on the impact of this frame selection? Even for a subset of one dataset, it would have been interesting to understand the breadth of potential with different static clips.

• Response to Q2: This is a very interesting point. We wanted to ensure that our method did not require any additional input or assumptions about what biases were occurring to maintain solid generalization and minimize computation. At first, this meant not having a method/model to choose frames that are more biased than others. However, this is needlessly pedantic, since choosing a frame position from within the clip takes minimal input and can have a drastic effect. As you say, it is interesting to evaluate the impact of different static clips. We explored frame selection strategies for static clips, evaluating first, middle, last, and random frame positions, shown in Table 1 below. Here is a summary of our key findings:

  • First and last frames sharply decreased performance, likely due to scene changes or irrelevant information before and after actions, even in the trimmed setting.
    • Contrasting the action learning from irrelevant information is trivial, not learning anything useful.
  • Random frame selection ensures variety in static objectives, leading to strong results.
  • Notably, middle frame selection improved performance over random selection!
    • This makes sense, since the middle frame is likely to contain the full background and actor in the middle of performing an action, making it a hard negative sample for the adversarial learning process.

  Using a sophisticated method to detect actors/backgrounds in frames (in trimmed or untrimmed setting) and choosing frames based on the existence of both in the chosen static frames would likely achieve the best performance, but this adds too much inductive bias and computation, so we avoid this in this work. Thanks for bringing this up, this experiment led to valuable insights that improved our overall results.

  Table 1: Experiment with choosing a specific frame for the static adversarial objective.

  Method	IID	SCUBA	SCUFO	ConflFG	ContraAcc
  Random Frames	73.20	53.22	0.42	49.84	53.02
  First Frame	72.75	50.92	0.18	45.59	50.91
  Middle Frame	72.81	53.53	1.50	48.13	53.22
  Last Frame	72.68	49.49	0.40	42.42	49.40

Q3: Wouldn't it be clearer to concretely specify that the action label notation is a one-hot vector? It is one-hot, right?

• Response to Q3: You are correct, this the label vector is technically a one-hot vector, we can specify this in the final version.

Q4: At line 218, shouldn't "maximized" be "minimized"? At least, something in that sentence does not match up: "p(t) is still matched to gt distribution y, but the similar ....maximized" Maximizing the "similarity" (a loose term here) is minimizing the ce loss.

• Response to Q4: Thanks for reading closely and for pointing this out, this is a crucial part to get correct in writing. As you say, the loss should be maximized, not the "similarity" (perhaps alignment is a better term). It should be more clear to say that the cross-entropy should be maximized in this paragraph.

Thank you for highlighting our strengths and for the comprehensive analysis of the paper.

W1: The paper is is somewhat "small" in that although it describes what seems to be a new method and its evaluation on reasonable benchmarks, there is little discussion around key elements of the method, their limitations, their rationale, etc. (see questions) Q1: The mechanism for adversarial learning in the paper is very specific to action recognition. How can this mechanism be generalized to be of broader interest to the ICLR community?

• Response to Weakness 1 & Question 1: We humbly request the reviewer to consider the significance of action recognition as a foundational problem in the broader domain of video understanding. Below, we provide both context and empirical evidence to demonstrate the broader applicability of our approach:

  • Importance of Action Recognition in Video Understanding:
    • Action recognition serves as the core problem for advancements in video understanding. Many state-of-the-art models in the diverse tasks of video understanding are trained on large-scale action recognition datasets (e.g., Kinetics-400) and subsequently adapted to diverse tasks. Even though standard action recognition models are trained on trimmed videos, their learned representations are directly transferable to various downstream tasks, often without fine-tuning (frozen pretrained model). Such tasks require an encoder to compute high-quality local action understanding, then slide this across videos and model global information across these sets of local low-dimensional features, instead of trying to pass in the high-dimensional videos all at once. In these scenarios, having an unbiased, powerful encoder for trimmed action recognition is crucial.
  • Empirical Evidence of Broader Applicability of our Method:

    • We provide empirical results in the table below to demonstrate the utility of our proposed debiasing training paradigm across multiple downstream tasks:

      • Anomaly Detection: The first downstream task we evaluate on is weakly supervised anomaly detection. The task requires a model to localize the frames within long, untrimmed videos where some anomaly (defined by the dataset) occurs. Due to the long videos, this task typically starts with a set of features extracted in sliding-window fashion from a Kinetics400 (or similarly) pretrained video encoder. In this task, discriminability between feature segments is crucial in order to localize anomalous segments. If an encoder exhibits a background bias for example, then it may output similar features for two clips with the same background but drastically different foregrounds, making the anomalous segments difficult to distinguish. For this weakly supervised anomaly detection task, we report results on the UCF_Crime [1] dataset as the frame-level Area Under the Curve (AUC) of the Receiver Operating Characteristic (ROC), which is the standard evaluation metric for this task. We use one of the current SOTA methods MGFN [3] with unchanged hyperparameters for this evaluation, only swapping the feature sets used to those extracted from a baseline model and a model trained using our framework on HMDB51.
      • Temporal Action Localization: The second additional task we evaluate on is untrimmed temporal action localization. The goal of this task is to identify time intervals where particular action classes occur within long video. Similar to weakly supervised anomaly detection, this tasks utilizes features extracted from a pretrained video encoder and stands to benefit from models with less static bias and better temporal modeling. For the temporal action detection task, we report results on the THUMOS14 [2] dataset. Evaluation is given as mean average precision (mAP). We use a SOTA model TriDet [4] with standard hyperparameters, again only swapping the feature sets used in a similar fashion to the UCF_Crime protocol.

    • In the table, HMDB51 (OOD) refers to the Contrasted Accuracy (Contra. Acc.) results explained in Main Paper Section 4.4. Table 1 (next comment, Part 2) shows all of these results. Notably, we see that performance is greatly improved across tasks that require high-quality temporal understanding. These results highlight that our approach is not only relevant for action recognition but also beneficial for diverse downstream tasks in video understanding, further establishing its broader impact.

Thank you for your positive comments!

W1: Are there more visual examples that can depict biases in action recognition?

• Response to Weakness 1: Absolutely, we can include more. Please download the updated supplementary material. We have added a video file titled paper7609_bias_examples.mp4, more qualitative examples can be seen there. Let us know if you would like to see any more.

W2: Whether the method proposed in this paper can be used for skeleton-based action recognition task？

• Response to Weakness 2: While we have not explicitly validated our methodology in a skeleton-based context, our core approach would definitely transfer. One potential bias in skeleton action recognition could be related to static poses, for example predicting a "throwing" action based on a single pose with one arm raised. Applying our static adversarial loss could mitigate this static pose orientation bias, encouraging the model to better consider the motion differences between skeletons. This could be an exciting future direction, and we plan to publicly release our code to support and encourage further exploration in this area.