ActFusion: a Unified Diffusion Model for Action Segmentation and Anticipation

[Inference setup in LTA]

We would like to clarify that our model can predict future actions with an arbitrary length by adjusting the number of mask tokens needed for prediction; our model itself does not require a ground-truth length of anticipation, and the ground-truth length is used to generate the same length of anticipation for the convenience of evaluation in testing. Note that previous methods [19, 24, 43] also commonly used ground-truth video lengths during inference to generate the final predictions, which can be found in the released original codes. We obtain the codes for [19] from the authors. We hope this clarifies the reviewer's concern. Otherwise, please let us know.

[Novelty]

Please refer to the general response for the novelty.

[Experiments on Assembly101]

Due to limited resources and time, we were unable to finish the full-scale experiment on time. In response to suggestions regarding Assembly 101, we instead did a smaller scale experiment by randomly sampling 10% of the entire training set to train the models and evaluate the performance on the full validation set to ensure a fair comparison.

Table R5 compares the performance of ActFusion and LTContext [5], the state-of-the-art TAS model on Assembly 101 from the experiment. We find that our model outperforms LTContext across all metrics, showing the potential advantages of the proposed method. We will provide a full-scale experiment in the final manuscript.

[Table R5. Experiments on Assembly101]

[Explanation of Figure 1(c)]

Please refer to the general response for a detailed explanation of Figure 1(c).

[Missing references]

Thank you for letting us know. We will include MVGA [R4] and RTK [R5] in our final manuscript.

[R4] B. Jiang et al. RTK: Video action segmentation via contextually refined temporal keypoints. In ICCV’23.
[R5] N. Aziere and S. Todorovic. Markov game video augmentation for action segmentation. In ICCV’23.

Thanks very much. I really appreciate the additional experiments. It is helpful to see the new results without using ground-truth length, which are quite reasonable.

It is surprising that all previous methods have utilised the ground-truth length during testing. I would flag this as a critical issue worth the community's attention. Revealing this issue using the above experiments might be a significant contribution to the community.

Therefore, I have greatly increased my score, on the condition that the additional experiments will be included in the later version. And all the other experiments should also be 'totally' updated to this no ground-truth version. However, this might be too substantial and there is no mechanism to guarantee that.

Overall, this is really a difficult case. Therefore, I have also changed my rating to the lowest confidence, and I would leave this to the chairs for the final judgement.

[W1: Using the ground-truth video length during inference]

Thanks to your response, now we fully understand the point regarding the use of the ground truth video length during inference. To address this concern, we conducted additional experiments where no ground truth length is used in testing; following your suggestion, in testing, we fixed the length of future frames (i.e., mask tokens) to the maximum number of future frames in the training set.

Experimental results in this setting on the 50 Salads dataset are reported in Table R6. In this table, the column ‘use of GT length’ indicates if each model exploits ground-truth length during testing. We found that, following the first paper introducing long-term dense action anticipation [2], all previous methods utilize the ground-truth length during testing, except for those whose codebases and/or inference setups are not available [25, 51, 65]; we marked these methods 'unknown' in the column.

In the table, ActFusion* represents our original model using a fixed number of mask tokens without retraining. To ensure a fair comparison, we applied the same testing scheme to Farha et al [19] (denoted by Farha et al*) and FUTR [24] (denoted by FUTR*), both of which originally utilize the ground-truth length during testing. In this setting, ActFusion* still outperforms Farha et al* and FUTR*, and as the reviewer expected, all of these methods perform worse.

This performance drop is due to the use of the ground-truth video length in training, which results in a discrepancy between the training and inference setups. To mitigate this issue, we retrained our model while fixing the number of mask tokens to cover the maximum number of future frames in the training set. The retrained results, presented in the last row of Table R6 and denoted as ActFusion†, achieve the state of the art in long-term action anticipation (LTA). We observed that fixing the number of mask tokens brings performance gain when the prediction ranges are relatively short, as the model benefits from more stable predictions. However, for longer predictions, particularly when the prediction ratio is set to 0.5, we observed performance degradation compared to ActFusion. This degradation is probably due to the fact that it becomes more difficult for the model to determine the end of an activity. Nonetheless, these results demonstrate that our method can be flexibly adapted to different numbers of mask tokens, ultimately achieving the state of the art in LTA.

We sincerely hope this clarification addresses your concerns. We believe that the experimental setup you suggested will contribute significantly to the field by providing more realistic and reasonable evaluation protocols, and we will include all of the above results in the revision.

[Table R6. LTA results with and without using ground truth length]

• In the table, the bolded values represent the highest accuracy among the models that do not use ground truth length, ensuring a fair comparison.

[Comparison with DiffAct]

Please refer to the general response to the comparison with DiffAct.

[Loss ablation studies]

We conduct ablation studies on the loss functions: boundary loss, smoothing loss, and cross-entropy loss. Table R4 presents the results, demonstrating that the combination of bounding loss $L_{bd}$ and smoothing loss $L_{smo}$ is effective for both TAS and LTA. While the effectiveness of these losses in TAS is well-documented in previous research [18, 64, 42], their impact on LTA has been less explored. Notably, the smoothing loss leads to significant performance gains in both tasks, indicating that smoothed predictions are beneficial.

[Table R4. Loss ablation studies]

(a) Results on TAS

(a) Results on LTA

[Computational efficiency]

Table R5 compares the computational cost of our model with ASFormer [64] for TAS, and FUTR [24] for LTA, in terms of the number of parameters, GPU memory usage during inference, and inference time. For TAS, as shown in Table R5 (a), although ASFormer has fewer parameters with lower inference time, it requires more GPU memory during inference and obtains lower performance. To improve computational efficiency, we reduce the DDIM inference steps to 10 and 1. This reduction decreases inference time while maintaining superior performance over ASFormer.

For LTA, as shown in Table R5 (b), our model is approximately eleven times smaller than FUTR, uses less GPU memory, but has a longer inference time. By reducing the DDIM inference steps to 1, our model achieves a similar inference time to FUTR. Overall, our model is practical and efficient since it can handle both TAS and LTA tasks with a unified model, eliminating the need for separate models and reducing GPU resource usage and the time required for separate training. Note that we use model checkpoints from the official GitHub repositories for all comparisons.

[Table R5. Computational efficiency]

(a) Results on TAS

(a) Results on LTA

[Explanation of Figure 1(c)]

Please refer to the general response for a detailed explanation of Figure 1(c).

We thank reviewer uA7n for the motivating feedback. We are pleased to hear that most of the concerns have been addressed by our rebuttal. The results discussed in the rebuttal will be included in the final manuscript.

[Novelty]

Please refer to the general response for our novelty.

[Marginal performance]

We would like to clarify that the performance improvements achieved by our model are significant. Figure R1 in the pdf file illustrates the performance of the Top 10 TAS models for each dataset listed in Table 1, based on their average performance across all metrics. The average performance gain between the adjacent models is 0.5 percentage points (pp), 0.7 pp, and 0.6 pp for the 50 Salads, Breakfast, and GTEA datasets, respectively. Our model, ActFusion, achieves performance gains of 0.7pp, 0.4pp, and 1.1pp compared to the second-best models for each dataset, and 1.1pp, 0.4pp, and 1.5pp compared to DiffAct. We believe the performance gains are meaningful, with notable improvements in two datasets.

[Reconstruction of masked features]

Masked auto-encoding is a technique used in training NLP models like BERT [R1] and has recently been adapted to vision models [21, 27, 68]. Inspired by this approach, we train our model to reconstruct input video features from the masked tokens as an auxiliary task. Specifically, we employ MLP layers on the encoder embeddings to reconstruct the input video features and apply mean squared error (MSE) loss $L_{\text{recon}}$ during training.

Table R2 shows the overall results on both TAS and LTA tasks. In TAS, overall performance increases. We conjecture that reconstruction helps the model gain a deeper understanding of the underlying data structure and temporal dynamics by predicting the missing parts of the input. In LTA, we find that reconstruction is more effective on relatively short-term anticipation. Since short-term predictions are often based on more immediate context, there is less uncertainty. As a result, reconstructing masked features helps the model capture immediate patterns and transitions more accurately. However, for long-term predictions, as the model attempts to predict further into the future, the uncertainty increases significantly. Long-term predictions involve more variables and potential changes, making them inherently less predictable. This increased uncertainty might cause performance degradation, making reconstruction less effective for action anticipation.

[Table R2. Effects of reconstruction loss]

(a) Results on TAS

(a) Results on LTA

[R1] J. Devlin et al. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL’19.

[More analysis on LTA]

Table R3 shows the LTA performance across different observation ($\alpha$) and prediction ($\beta$) ratios. Average anticipation performance improves as the observation range increases. With more observations, uncertainty about future actions is relatively reduced, leading to more accurate predictions. Conversely, average anticipation performance decreases as the prediction range increases. Predicting further into the future presents more challenges due to increased uncertainty, as future actions become less predictable and more variable.

[Table R3. Analysis of observation and prediction ranges in LTA]

[Extension to the open-set action recognition]

We appreciate the reviewer’s insight on extending our work towards open-set action recognition. To achieve this, we can use frozen image and text encoders from CLIP [R2] to obtain shared representations for actions and text embeddings. Similar to [R3], these embeddings can then be integrated into our model to enable open-set action recognition. We plan to explore this as a future direction for improving our approach.

[R2] A. Radford et al. Learning transferable visual models from natural language supervision. In arXiv’21.
[R3] D. Chatterjee et al. Opening the vocabulary of egocentric videos. In Neurips’23.

[Reasons: segmentation is more helpful on anticipation]

We find that segmentation greatly enhances anticipation, while the effect of anticipation on segmentation is less significant (L271-272). Segmentation directly improves anticipation by providing accurate contextual cues and action boundaries of the observations, enabling the model to make more precise future anticipation. In contrast, anticipation helps segmentation more indirectly. Anticipation encourages the model to consider long temporal relations of actions within an activity, which may not result in immediate performance improvement in segmentation.

[Explanation of Figure 1(c)]

Please refer to the general response for a detailed explanation of Figure 1(c).

[Multiple underlined values in Table 2]

Thank you for pointing out. The underline of the performance of DiffAct on the 50 Salads dataset will be removed.

Dear reviewer FW3D,

We'd like to thank again for your effort and time dedicated to our submission. We've addressed your concerns in our rebuttal, and it would be very helpful if you could give us any further thoughts and update your scores before the author-reviewer discussion phase ends. Your opinion would be invaluable to us in improving our work, and we would be glad to respond further to your questions. Thank you for your consideration.

Best regards,
Authors