OnlineTAS: An Online Baseline for Temporal Action Segmentation

We thank the reviewer for finding our contribution claims to the meaningful online TAS task modest and accurate. We address the reviewer's comments below.

Weaknesses

---

W1. Inference speed

A: Runtime evaluation, conducted on our Intel Xeon Gold 6442Y (2.6 GHz) and a single Nvidia A40 GPU, is reported in time (ms) per frame / FPS in the table below. Note that the inference speed is identical for online and semi-online modes due to their identical input length.

[*]: NVOFA SDK. https://developer.nvidia.com/opticalflow-sdk

These are our findings:

1. Our architecture achieves an inference speed of 238 FPS with precomputed I3D features.

2. Feature extractions for both RGB and optical flow inputs are fast, reaching at least 3x the real-time rate of 25 FPS.

3. The TV-L1 algorithm used by I3D, not optimized for online requirements, supports 6 FPS. However, Nvidia Optical Flow SDK (NVOFA) with hardware acceleration can boost this speed to 714 FPS.

In conclusion, to use I3D features, the optical flow calculation is the main bottleneck. Real-time application is feasible with more advanced acceleration algorithms.

---

W2. Results on Assembly101

A: Due to the limited time, we could only experiment with a subset of a single view (C10095). Results are shown in the table below. Our approach can also effectively achieve reasonable performance compared to the offline setup. The pure online for the Assembly101 dataset is very challenging in terms of segmental metrics (row 2), while our proposed post-processing significantly boosts segmental performance.

---

W3. Contribution claim

A: Thanks for pointing out these concurrent works. After a close examination, O-TALC was released on Arxiv in mid-April indicating acceptance as a short and unindexed paper in TAHIR; while Progress-Aware Online Action Segmentation only became available at CVPR one month after our submission. We thank the reviewer and will revise our statement in our updated manuscript.

We thank the reviewer's acknowledgment of our response to the weaknesses and their prompt feedback.

To clarify, “online” refers to processing data incrementally as it becomes available. It does not necessarily imply zero latency (i.e., real-time processing). Many existing video understanding methods which claim to be (and are accepted) as online do not work in real-time. For instance, (semi-)online video segmentation methods such as GenVIS [1] operate at 6 FPS and DEVA [2] at 6.6 FPS; while online action detection methods like Testra [3] at 12 FPS, MAT [4] runs at 8.1 FPS, E2e-load [5] at 8.7 FPS, and MATR [6] at 6.0 FPS. Among these, [3,4,6] are not “real-time” because of their optical flow extraction.

Additionally, we have included the speed of the CUDA version of the TV-L1 for optical flow calculation, using the same hardware configuration as we mentioned, in the updated table below. With the CUDA version, our apporach can achieve a real-time inference speed up to 33 FPS. We believe that utilizing the cuda version of the same TV-L1 algorithm will have a minimal impact on the overall performance.

We hope this clarification addresses the reviewer's concern regarding the inference speed of our approach.

[1] A Generalized Framework for Video Instance Segmentation. CVPR 2023

[2] Tracking Anything with Decoupled Video Segmentation. ICCV 2023

[3] Real-time Online Video Detection with Temporal Smoothing Transformers. ECCV2022

[4] Memory-and-anticipation transformer for online action understanding. ICCV 2023

[5] E2e-load: End-to-end long-form online action detection. ICCV 2023

[6] Online Temporal Action Localization with Memory-Augmented Transformer. ECCV 2024

We thank the reviewer for acknowledging the task importance and our effective presentation of the work.

Weaknesses

---

W1&2. Novelty on memory bank, semi-online inference, self- and cross-attention

Thanks for your comments. Indeed, attention and memory banks are common architectural components and semi-online inference is also an established concept. Note that we don’t claim the memory bank, attention mechanisms and semi-online inference as our technical contributions.

The distinction and novelty lie in how the objective and design are tailored to the specific task rather than the architectural components or abstract concepts stand-alone. Our focus is centered on the unique integration of the memory bank, attention mechanism, and semi-online inference for the online TAS task.

Here are detailed distinctions:

Memory bank

In [1,2], the memory design supports instance matching where each memory token represents object instances and is selectively updated via feature similarities [2]. While our memory design accumulates contextual information for feature enhancement, storing contexts at different granularities: clip-wise coarse long-term memory and frame-wise fine-grained short-term memory. Our memory capacity adapts over time to retain as much contextual information as possible.

[1,2] update Q directly based on K and V, whereas ours retrieves relevant memory for the query clip with Trans. Decoder and then enhance features with SA and CA.

In addition, our method shows over a 30% increase in Acc (Table 9) compared to existing work like LSTR [44], using similar attention and memory techniques.

Semi-online Inference

Thank you for pointing out [3], semi-online inference is not a claimed contribution but is presented to offer a more comprehensive evaluation due to the clip-based nature of the model's inputs and outputs.

In summary, as stated in L43-49, our technical contributions are:

1. Addressing the novel online TAS problem;

2. Developing CFA leveraging adaptive memory;

3. A fast and effective post-processing for over-segmentation in the online setup.

---

W3. Trans. Decoder necessity

A: We include a Trans. Decoder is to gather context information specific to the current clip, which we then use to enhance the clip features for better online segmentation.

The ablation results above (on 50Salads) indicate that performance decreases when the component is removed and memory is directly fed as K and V for CA (row 2). Further removal of the memory (row 3) leads to an even greater performance drop, highlighting the importance of the context information in the online task.

---

W4. Overlap semi-online performance

A: The table below shows that with a stride of $\delta=64$ and a window size of $\omega=128$, overlaps lead to better results than the frame-wise online mode but lag behind the non-overlapping mode.

Overlapping results are affected by multiple predictions with various contexts, while the non-overlapping case benefits from a shared context, leading to better accuracy and less over-segmentation.

---

W5. Acc decreases after post-processing

A: Segmental metrics and frame-wise accuracy are not well-aligned in TAS. A high accuracy doesn’t necessarily ensure less over-segmentation and vice versa. Consider the extreme case where every other frame is interspersed with the wrong action, the segmental measures are very poor, but the accuracy is still 50%. This trade-off is also observed in [R1].

Our post-processing prioritizes reducing over-segmentation over frame-wise accuracy. Fig. RA in the rebuttal pdf (sample example as Fig. 3) shows post-processing: 1) removes fragments (black boxes). 2) may reduce Acc, particularly at action boundaries (red boxes).

[R1] Unified Fully and Timestamp Supervised Temporal Action Segmentation via Sequence to Sequence Translation, ECCV 2022.

---

W6. Performance gap between online and offline

A: We respectfully disagree that a performance gap between online and offline models is a weakness.

Such a gap is well-expected and is directly attributed to the online nature, in which future temporal context is not available for making predictions. In a similar fashion, [R2] found that under the same offline setup, performance drops when the input window size is reduced and cannot account for sufficient context.

[R2]. How much temporal long-term context is needed for action segmentation? ICCV 2023.

Questions

---

Q1. Why use offline models?

A: We thank the reviewer for the question.

Successfully solving TAS requires merging visual features over long temporal ranges of context, while preserving local semantics. Existing offline models are designed specifically for this task and excel at it, making them a natural starting point rather than building from scratch.

Additional advantages include: 1) using the same backbone to facilitate performance evaluations and comparisons to the offline model and 2) analyzing specific performance discrepancies to identify meaningful areas for improvement.

We thank the reviewer for recognizing the novelty of our CFA module for the online TAS problem and the effectiveness of the post-processing for mitigating the over-segmentation issue.

Weakness

---

W1&4. Runtime analysis and computational requirements

A: We thank the reviewer for the comment.

Runtime

Kindly refer to our response to W1 for Reviewer aGP1.

Computation requirement

We compare GPU memory requirements for inference between offline and online models. Offline models like MSTCN and ASFormer use 551M and 716M per every 10,000 frames, respectively. Note that the memory requirement will scale linearly with the actual video length as these models operate on the full sequence. In contrast, our model has a fixed memory requirement of 1.2G for each inference, regardless of the actual video length. This facilitates its application in dealing with online streaming videos that can be infinitely long.

---

W2. Error analysis.

A: Thanks for the suggestion. We show some qualitative examples in Fig. 3, and we will add more to the revision. For failure cases, we have the following two observations:

1. Action start often delays due to the need for more frame information to predict new actions, especially when facing semantic ambiguities at action boundaries.

2. Persistent over-segmentation happens when the network makes incorrect but confident predictions, which could be improved with a stronger backbone or better temporal context modeling.

---

W3. How memory scales with video lengths

A: We thank the reviewer for the question. When the memory capacity is exceeded, our method discards the earliest memory in a FIFO manner.

Our ablation study (Table 5) includes scenarios where the video length exceeds the memory limit. For memory sizes of 16, 32, and 64, the earliest memory are discarded as the average length of 50Salads is ~5.8K frames. With the size reducing, the performance gradually drops and reaches the lowest of 79.8 in Acc compared to the peak of 82.4. Note that with the memory size set to 16, our approach only retains long-term information from up to 192 frames, 30x less than the average video length.

---

Questions

Q1. Different feature extractor and backbone

A: Thanks for the suggestion. We experiment on three common TAS datasets replacing the MS-TCN backbone with ASFormer and report the results in Table RA in the rebuttal pdf. Our approach is still effective in boosting online segmentation performance.

Unfortunately, we weren’t able to conduct the experiments with other optional feature extraction backbones due to the time and resource constraints for the rebuttal. We will add this to the camera-ready.

Q2. Online setup under other forms of supervision

A: Thanks for the interesting question. Weakly- and unsupervised approaches typically involve iterative model training and label estimation. Our post-processing technique, as a label refinement process, may be used in such setups to refine network prediction and generate temporally consistent pseudo-labels. The two lines of work are orthogonal and serve as interesting thoughts for future work of addressing the online problem.

---

Q3. Hyperparameter sensitivity

A: These studies are already reported in Table 5. Our approach is reasonably tolerant to the clip size and memory length change. For example, the performance only drops 1% when the size is reduced to 1/4 (32) of the original default (128).

---

Q4. Integrating post-processing for training

A: That is a good question. The post-processing is to refine the frame labels predicted by the model; we can see its application in weakly and unsupervised setups to refine pseudo labels as we explained in response to W2. However, as our current approach is fully supervised, there’s no need for label correction during training as the ground-truth labels are provided.

---

Q5. Latency accuracy trade-off analysis between online and semi-online modes

A: The inference speed for one pass is identical for both modes as their input sizes are the same. However, their latency is different. Online inference operates on a frame basis and the latency is dependent only on the inference speed while semi-online mode requires additional latency to gather frames up to the clip lengths (128 frames at a standard 25 FPS corresponds to 5.12 seconds).

In terms of performance, the online approach is less competitive than the semi-online with an average gap of around 2-5% (see Table 1). This is likely because of the better preservation of temporal continuity in the semi-online setup, which we discussed in L.219-222.

To summarize, online inference has a better real-time response than semi-online inference. Semi-online inference achieves better performance. The choice of these two modes will depend on the application priority, if the real-time inference is required, online would be preferred, if accuracy is preferred and the task is less time-sensitive, then semi-online is suggested.

We thank the reviewer for the constructive question and will include these discussions in our updated manuscript.

---

Limitations

L1. Potential challenges in diverse and real-world streaming cases

A: Thank you for your suggestion. Handling diverse and real-world videos presents several challenges.

One common scenario involves interrupted actions, where a subject abruptly switches to a different action, leaving the ongoing action unfinished. These interruptions can be challenging for the model to handle effectively.

Additionally, the extended length of the video poses another challenge. Streaming videos can be infinitely long, so effectively managing and preserving long-form history within a fixed memory budget becomes a critical issue.