Efficient Temporal Action Segmentation via Boundary-aware Query Voting

We thank the reviewer for investing time in reviewing our work and acknowledging our contributions.

1. For Weakness 1

(utilizes existing frameworks) We understand the reviewer's concerns regarding our use of existing frameworks; however, we employ them solely to achieve the desired functionality. An important aspect we need to clarify is that we focus on how to adopt a new pipeline, i.e., query-based pipeline, into efficient TAS rather than some specific network design. To achieve more efficient TAS, we experimentally find that query-based processing can reduce dense information, thus reducing FLOPs. Therefore, we adopt query-based prediction from a DETR-style model to improve the running time. This lets us establish a new paradigm in TAS, different from the frame prediction in previous work. Our main effort is how to adopt the DETR-style model into TAS to achieve the expected performance, rather than to design some specific network structures. Specifically, we focus on introducing functional modules with effective organization, i.e., frame-wise encoder and decoder, transformer decoder. These modules can be implemented with any network. However, in our paper, as acknowledged by the reviewer, we employ existing frameworks to achieve these functional modules. This approach allows us to: 1) fairly compare with previous methods by sharing the same backbone, and 2) verify the effectiveness of our pipeline, which relies on the new paradigm rather than network design.

Furthermore, although these existing frameworks have been successful in previous works, they may not be effective in our new pipeline, which is organized with different functional modules. Therefore, in terms of TAS tasks, we have made numerous efforts to optimize our approach. These efforts include experimenting with different matching strategies, establishing connections between frame decoders and transformer decoders, and implementing auxiliary loss functions. We analyze these aspects in detail in the main paper and supplementary materials and hence provide insights for designing a superior pipeline.

It is worth mentioning that there are many impactful works to adopt the DETR-style model into different fields as what we do in TAS tasks, e.g., TrackFormer[1] for multi-object tracking, MaskFormer[2] for image segmentation and CLTR[3] for crowd localization. However, our work is the first such exploration for TAS.

[1] T. Meinhardt, A. Kirillov, L. Leal-Taixe, and C. Feichtenhofer. Trackformer: Multi-objecttracking with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8844–8854, 2022.

[2] B. Cheng, A. Schwing, and A. Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances in Neural Information Processing Systems, 34:17864–17875, 2021.

[3] D. Liang, W. Xu, and X. Bai. An end-to-end transformer model for crowd localization. In European Conference on Computer Vision, pages 38–54. Springer, 2022.

2 .For Weaknees 2

(Running Time) Thank you for the suggestion to include detailed results to improve our paper. Based on the reviewer's suggestion, we have added the table below. And find that the original module's additional running time is due to the NMS post-processing. To ensure a fair comparison, we have listed both the forward time and the post-processing time below. Thanks again for the reminder and we will update the manuscript.

Thank you for the valuable suggestions to enrich the analysis of our model, most of which are included in the existing supplementary. It looks like you didn’t find the supplementary material -- we double-checked and it’s not a problem, and other reviewers haven’t had this issue either. Please inform us or contact the conference organizer/AC if there are any issues with locating or downloading the supplementary material. We refer to the location of the following question in our supplementary material.

(Weakness 1 and Question 1: more ablation studies and query tokens) We agree that more ablation studies help us to analyze the model further, and we include some of them in supplementary, such as the number of query tokens, single/multi-level feature connections, auxiliary loss, and other factors. Specifically, as for the number of query tokens, the experiment indicates that a slight increase in the number of queries can enhance the model’s ability without significantly raising computational costs. However, the model’s performance peaks at 100 queries, achieving the highest accuracy of 89.5% at a computational cost of 4.45G FLOPs (more details can refer to Table 2 of the supplementary material).

(Weakness 1 and Question 2: a detailed comparison of the different loss functions) In supplementary Table 3, we present the performance results when the auxiliary loss is included under single or multiple features. Notably, applying auxiliary losses in multi-level features yields a substantial improvement in accuracy (2.1%), surpassing the 0.5% accuracy gain observed in single-level features. This suggests that auxiliary losses have a more pronounced effect on multi-level feature representations.

(Weakness 2 and Question 3: qualitative results) Qualitative results, which can help us to analyze segmentation results, especially among the time dimension. So, in the supplementary material, we also provide additional qualitative results in Figure 2 to demonstrate the efficacy of our proposed query voting. In the 50Salads dataset, we present enhanced visualizations for a more comprehensive analysis. The figure includes the outcomes of instance segmentation, the segmentation results without boundary integration, and the ground truth.

(Weakness 3: model size compared with DiffAct)We apologize for the concerns and recognize that we should clarify that efficiency refers to shorter running time. Thus, despite DiffAct's smaller size, its running time is longer than ours. DiffAct is a multi-stage method based on the diffusion model. It is built on a model with a small size but requires 25 iterations to refine the results, significantly increasing the running time. By contrast, our BaFormer requires only 6% of the time used by DiffAct and obtains a comparable performance. Directly comparing single-stage methods with multi-stage ones is somewhat unfair, we therefore divide them into two groups in Table 6. Alternatively, we can adjust the models to have similar running times for a fair comparison, we present the results in Table 7. Our method shows superior performance, achieving an accuracy of 89.5% and F1@10 of 89.3, compared to DiffAct's accuracy of 74.2% and F1@10 of 48.3 (more experimental results with different steps for DiffAct are shown in Table 4 in the supplementary material). Furthermore, we can extend our method to a multi-stage one to increase the accuracy to 90.5%, which surpasses the original DiffAct. However, in this paper, we focus on improving the performance of single-stage methods rather than relying on multi-stage methods.

(Weakness 4 and Question 4: more analysis of hyperparameters) The supplementary material includes an analysis of hyperparameters, such as the number of the query tokens(Table 2), the connection between the frame-wise and transformer module (Table 3), the transformer decoder layer(Table 1), the single or multiple features (Table 3) and so on.

(Limitation) The limitation is also discussed in the supplementary. Our proposed approach is under the query-based Transformer framework, which is well-known for its slow training convergence and data thirty. We observe that training query-based Transformer converges slower than training frame-wise approaches. For the discontinuous binary mask predictions, we assume that it may also be attributed to the limited data of action segmentation benchmarks. We hope that these factors can inspire future work.

We thank the reviewer for acknowledging our contributions and below is our response regarding the reviewer’s concerns.

1.For Weakness:

However, given the design of voting between frame-wise and segment-wise predictions, one expected advantage of having access to segment prediction is to reduce the over-segmentation issue, yet the effect of such voting scheme between frame-wise and segment-wise predictions did not achieve the expected effect. On most of the metrics, the propose approach did not manage to outperform existing SOTA approaches, for example DiffAct, especooally on the segmental metrics.

Response:

We apologize for not providing a clear note on how to compare our performance with others, despite having separated the methods into single-stage and multi-stage in Table 6. Since our work focuses on the efficiency aspect of TAS, it is somewhat unfair to compare the single-stage (i.e., BaFormer) and multi-stage (i.e., DiffAct) models on segmental metrics. That is why we categorize all methods into multi-stage and single-stage groups, as shown in Table 6. Specifically, multi-stage models involve many stages of refinement, which are time-consuming. Our single-stage model is designed to be more efficient. Although our model shows a 0.5% decrease in F1@10, we improve accuracy by 0.8% on the 50Salads dataset, and notably, our model requires only 6% of the inference time compared to DiffAct. DiffAct, based on a diffusion model, includes 25 inference steps, significantly increasing the time required to refine segmental metrics. When comparing models with similar inference times, DiffAct achieves 64.9% on F1@10 and 88.6% accuracy, while our method achieves 89.3% on F1@10 and 89.5% accuracy, demonstrating superior segmental metrics of our model. Further details can be found in Table 11 of the supplementary.

We can extend our methods to multi-stage to improve the segmental metrics further; however, experimental results indicate that this increases processing time, which contradicts our primary objective in this paper—to design an efficient TAS model, as acknowledged by the reviewer. To demonstrate the effect of our model, we also extended it into a multi-stage framework and modified the visual features for accuracy 91.2% and 90.8%, which are superior to DiffAct.

To fairly assess the effectiveness of the voting mechanism in reducing the over-segmentation issue, we need to compare it within single-stage models. These models all include one stage with post-processing, and comparisons should focus on different processing techniques with similar backbones. For instance, compared to UVAST with FIFA, our method demonstrates superior performance in segmental metrics (88.9% vs. 88.3%), accuracy (89.5% vs. 84.5%), and faster inference (0.139s vs. 1.765s), which result from the query-based model design. This demonstrates that our approach has already outperformed existing SOTA approaches for the one-stage model.

2. For weakness of confusing notation in Sec 3

(About decoder) Sorry for the confusion. It should be noted that the encoder is frozen in all previous work for fair comparison, mentioned in the experiment of the paper. Thus, the decoder aims to further process features from the encoder into high-level information, making them more adaptable to the TAS task through training. We call it a decoder to distinguish it from the frozen feature encoder/extractor. The decoder architecture can be any network adopting from the existing work such as MSTCN and Asformer.

($C$ in exps) In the model, all instances of $C$ are set to 64, which is the feature dimension shared by different features. The value can be found in L217 of the experiment settings with "an output dimension of 64". We will clarify this by changing the phrase to "an output dimension $C$ of 64".

(Notation clarity) Thanks for the suggestion and we will add improve the clarification and eliminate the notation inconsistency in the revision. Specifically, $$ P_i^m = \varphi_m(Q_l, F_d) $$ represents the $i$-th mask features. Here, $m$ is the abbreviation of "mask".

We have reviewed all the notations and will use superscripts to represent the type of the variable and subscripts to represent the index. We will update the following notations:

Furthermore, for better understanding existing notations, we will add the following descriptions:

3. For Question 1

(Advantage of extra branch). As our approach is based on voting to yield segmentation results, the extra branch for boundary generation is essential for our method. The extra branch can generate high-quality boundaries without much additional effort (nearly 5% Flops assumption). It can bring a 2.3% improvement in accuracy and a 2.1% improvement in Edit score, compared to a direct summary of all query predictions without an extra boundary prediction branch.

4. For Question 2

Refer to the answer of above part "2. For weakness of confusing notation in Sec 3" .

We thank the reviewer for providing us with the opportunity to conduct a more detailed analysis of the performance on segmental metrics.

1.Respond to detailed table and gains

(Detailed table ) As the reviewer suggested, the answer is "Yes"—the weaker backbone, which we adopt for fair comparison, is one of the reasons for the underperformance in segmental metrics. Therefore, the detailed results of adopting another visual encoder, TSM[1], in the backbone to achieve superior performance are as follows:

This demonstrates that a stronger backbone can improve segmental metrics, with F1@10 increasing by 1.4%, Edit by 1.8%, compared to a weaker backbone.

[1] J. Lin, C. Gan, and S. Han. Tsm: Temporal shift module for efficient video understanding. In Proceedings of the IEEE International Conference on Computer Vision, 2019

(Performance gains)The answer is "Yes" to the reviewer's assumption of the performance gains corresponds to the above response to Q1. To provide more details on where the performance gains come from, the reviewer's suggestion to present the performance of query-based instance segmentation, both with and without the addition of the boundary branch, is highly valuable. The results are as follows:

2. Respond to the notation

(Notation) We appreciate the reviewer's efforts to help us in clarification. The second notation table is correct but little confusing; both the $**p**^m$ and $**P**^m$ exist, and their relationship is described in L156 of paper, $**P**^m = (**p**^m_i)_{i=1}^M$. To clarify, we assign the $l$ as the index for the layer of the transformer decoder, while the $i$ is the index for the query. Then the above notation table can be modified to:

3. Respond to comparison

(comparsion)In the paper, shown in figure 1, most of two-stage and one stage methods cannot achieve a good trade off between effectiveness and efficiency. Thus, we categorize all the methods into two classes to see what achievement we have for temporal action segmentation. In addition to Table 6, we also consider recent multi-stage methods, adjust them to have similar running times, and present the results in Table 7. These results demonstrate that we achieve better performance when our method operates with similar time constraints to the multi-stage methods. Furthermore, a more detailed comparison with multi-stage methods across different numbers of stages is included in Table 4 of the supplementary material.

Thank you for your comments. We provide discussions and explanations about your concerns as follows.

1. For Weakness1

(Query voting is very simple) We thank you for acknowledging our contributions to Matching Strategies and Voting. Regarding your concern about the simplicity of the process, we intentionally designed it to be simple to contribute to efficiency. In our paper, we aim to design an effective TAS model with high efficiency. So when designing such a model, we pursue two key points: (1) a highly efficient model structure to avoid heavy multi-stage or multi-step architecture; (2) integration of simple but effective post-processing upon high-quality model predictions. Specifically, our model transforms frame-wise predictions into query class-mask predictions for TAS. At the same time, we put much effort into exploring different matching strategies and extensively ablate different components (number of decoder layers, query quantity, auxiliary loss, global query for boundary generation, etc., as reported in experiments and supplementary material.) of the network architecture. After achieving high-quality query class-mask predictions as well as precise boundary generation, we are then enabled to use very simple but highly efficient query voting to realize the expected efficient TAS.

We can also adopt more complicated post-processing, such as the Viterbi algorithm, activity grammar, and so on. However, previous papers have verified that they are time-consuming, which is not what we desired. For example, in another single-stage method UVAST, when using the Viterbi algorithm it spends more than 480s with an accuracy of 87.4% on 50Salads dataset; while when using simpler post-processing, i.e., alignment decoder, it takes 0.577s but with a degraded accuracy of 79.5%. By contrast, our BaFormer employs simple voting with 0.139s and achieves a higher accuracy of 89.5%.

(BaFormer follows previous work) As for the concern that BaFormer follows previous work, BaFormer is designed to improve efficiency, and we experientially find that a query-based model can change the dense information into sparse information to reduce FLOPs. So, we follow the DETR-style pipeline to predict the query-based results. It gives us insight to take a new paradigm in TAS. Our focus is on how to innovatively adopt the DETR-style model into TAS that has not been fully explored by such an approach. Like BaFormer, many other promising works also follow the DETR model to achieve their different targets, for example, MaskFormer in image segmentation, CLTR in crowd localization, TrackFormer in multi-object tracking, and so on. They design new pipelines similar to DETR for different tasks as we do.

2. For Weakness 2

(Other visual encoder) Thank you for the suggestion to adopt other visual encoders, which will enrich our model's performance. Accordingly, we conducted experiments with TSM visual encoders compared with I3D. The results on 50Salads are as follows:

The results indicate that a superior visual encoder can further enhance performance.

3. For Weakness 3

(Terminology) Sorry for the confusion. In our study, the term "CNN backbones" refers to the single-stage MSTCN, and the "Transformer backbone" refers to the single stage of ASFormer. In "Frame-wise Encoder-Decoder", the structure of the frame-wise encoder is fixed as I3D for all methods in Table 7. Therefore we focus on different structures of the frame-wise decoder. As acknowledged by the reviewer, the term "backbone" specifically refers to the frame-wise decoder. We apologize for any confusion and thank the reviewer for pointing it out. For clarity, we have replaced the term "backbones" with "frame-wise decoder".

4. For Weakness 4

(Typo in reference) Thank you for pointing out this embarrassing typo. We promise to fix it and check carefully other references in the revision.

5. For Weakness 5

(Visualization in Figure 6) Again, sorry for the confusion, and we will add relevant clarification in the revision. Figure 6 comprises two parts. Specifically, to generate the top part, we binarize the predicted masks $\in R^{n\times T}$ to 0 or 1 with a threshold 0.5, where $n$ is the number of queries, and $T$ is the length of the video. Thus, the positions with a value of 1 indicate the presence of actions corresponding to the query class. Next, we apply different colors to these positions and each color represents a single action. Then query masks are stacked to form the top part. In the bottom part, we present video segment results of frame-based and query-based voting, along with the ground truth. The red arrow points towards a specific query within a segment proposal, illustrated between two vertical black dashed lines. The red dashed box shows the segment results within the proposal.

(Performance on segments with different lengths) Thanks for the suggestion. We conducted experiments with different segment lengths and the results are summarized below. We find that the performance of frame-based methods is slightly better on shorter segments, and the accuracy is shown below:

Thank you for the valuable feedback and the opportunity for further discussion.

1.Respond to I3D or TSM

(I3D or TSM) We understand the reviewer's concerns regarding the results under different backbones. To ensure a fair comparison, all methods in Table 6 of the paper use I3D features. Specifically, for all methods, TAS tasks use fixed I3D features rather than TSM features as input. We believe it is fairer to use I3D as the visual encoder, even though TSM features may yield superior performance.

2. Respond to shorter segments

(For shorter segments) Thank you for the valuable suggestion, which has helped us analyze methods more deeply across different segment lengths. We will include additional analysis based on the table presented for Question 5. This type of case arises from mask prediction in the query-based method. In binary mask prediction for shorter segments, the ratio between foreground and background is smaller than in longer segments, leading to varying accuracy in mask prediction across different segment lengths. This is similar to the challenge faced in small object detection using DETR. We will include this observation in the limitations of the manuscript.

To improve this case, potential solutions could involve reweighting the ratio of positive to negative samples, increasing the weight assigned to the mask prediction heads, and so on. This approach could guide our efforts in future work.

3.Performance on GTEA

(GTEA)We acknowledge the reviewer's concern regarding the performance on GTEA. While our method results in a 0.7% decrease in F1@10, it improves accuracy by 3%. This outcome is likely related to the shorter segment lengths in GTEA as described in part “for shorter segments”. We have provided data statistics across different datasets, where "Ratio" refers to the ratio between segment length and video length.

We observe that the ratio in GTEA is only 1/3 of that in 50Salads or Breakfast. Beyond accuracy, for efficiency, we achieve a running time of approximately 1/5000 of UVAST's time, resulting in a significantly faster TAS.